diff --git a/TEST-REPORT.md b/TEST-REPORT.md index fac348e..494aa21 100644 --- a/TEST-REPORT.md +++ b/TEST-REPORT.md @@ -70,9 +70,70 @@ Agent path works (v0.74.0), so these ran through the real controller→agent pip | Low-space backup behavior | SKIPPED | tied to C4 (filling 100+ GB impractical/unsafe unattended) | Bonus across the phase: **single-flight** is robust — observed rejecting concurrent restore (mutex), concurrent whole-guest backup (C7), and Tier-2 vs DB-dump. -## Phase 5 — Host reboot (optional) — _pending_ +## Phase 5 — Host reboot (optional) — **SKIPPED (unattended risk)** +A host reboot of the N100 is the highest-risk action unattended — if it doesn't POST/return (BIOS +AC-recovery etc.) the demo is down until physical access, which CC doesn't have. Deferred to supervised +(consistent with campaign #1). Guest-reboot recovery is proven (C8 + campaign #1). --- -## Feature matrix — _filled at end_ -## Cleanup confirmation — _filled at end_ -## Deferred / needs supervised — _filled at end_ +## Feature matrix +| Area | Verdict | +|---|---| +| PBS backup + verify + leak-fix gate | PASS | +| App deploy (HDD-bound navidrome) + redeploy race | PASS | +| Storage: scan/health/durable-id | PASS | +| Storage: label edit/revert | PASS | +| Storage: disconnect↔reconnect (4-state) | PASS | +| Storage: decommission↔re-enroll (non-destructive) | PASS | +| Storage: destructive format on loopback | BLOCKED (agent device-whitelist) | +| Chaos C1 kill app → restart | PASS | +| Chaos C2 kill controller | PASS recovery / 🔴 no auto-heal | +| Chaos C3 stop agent (graceful + drain) | PASS | +| Chaos C5 remove app.yaml | PASS | +| Chaos C6 eject drive w/ apps | PASS recovery / finding (recreate noise) | +| Chaos C7 kill mid-quiesce → Recover | PASS (crash-safety proven) | +| Chaos C8 reboot during op | PASS | +| Backup: concurrent mutex | PASS | +| Backup: stopped-stack | PASS | +| Single-flight (backup/restore/migrate) | PASS | +| `/api/backup/snapshots` | 404 (gap, carried from #1) | +| restore DB import additive | confirmed (gap, carried from #1) | +| C4 disk-full / low-space | SKIPPED (impractical) | +| Phase 5 host reboot | SKIPPED (unattended risk) | + +## Findings (new this campaign) +1. **🔴 docker restart policy not honored** — `unless-stopped` containers don't auto-restart after a + crash (`docker kill` → `RestartCount=0`, dockerd `stopping restart-manager`; Docker 29.5.3). **No + docker-level self-heal for crashed apps OR the controller.** The controller detects + a UI/operator + restart recovers apps; but a crashed **controller** has **no automatic recovery** (oneshot bootstrap + + dead restart policy) — manual `systemctl restart felhom-controller-bootstrap.service` needed. + *Recommend:* a controller watchdog (e.g. bootstrap `Restart=always` monitor or a systemd unit + watching the container) + investigate why the daemon's restart-manager is cancelled. +2. **Disconnect-recreate race (C6)** — disconnecting a drive-with-apps triggers the controller's + userdata-belt/recreate to try bringing drive-backed apps up *while the drive is absent* → + `mkdir …/userdata: permission denied` + transient `Created` states, instead of a clean held + missing-storage stop. Reconnect recovers it. *Recommend:* gate ensure-userdata/recreate on + "drive is a live mountpoint." +3. **FileBrowser skeleton on unmounted path** — disconnect/decommission log + `mkdir …/userdata: permission denied` from the FileBrowser sync on the just-unmounted path. Cosmetic + but noisy; gate on live-mountpoint. +4. **Agent device whitelist** refuses loopback (`/dev/loop0`) — good security, but means a loopback + can't be the destructive-disk scratch target; destructive format/migrate need a real spare disk or + a supervised run. + +Gaps carried from campaign #1 (still open): `/api/backup/snapshots` 404; restore DB import additive. + +## Cleanup confirmation +- No throwaway guest created this campaign (DR was campaign #1); scratch loopback torn down (`losetup -d` + rm). +- navidrome (test app) removed → back to **25 containers** baseline; uptime-kuma restarted healthy. +- All chaos breaks recovered + verified (C1–C8); agent **active**; felhom-flash + felhom-usb **re-enrolled** + mounted. +- All timed guards killed (0 `sleep` guards); test forwarder + host temp files removed; **0 loopbacks**. +- Notifications untouched this campaign (none configured); no password changes. +- **9201 running, 25 containers, 0 unhealthy; controller + cloudflared healthy; agent active.** + +## Deferred / needs supervised +- **Controller-recovery watchdog** + docker restart-policy investigation (finding #1) — highest priority. +- Destructive storage **format/migrate** on a real spare disk (loopback blocked) — supervised. +- **Host reboot** resilience (Phase 5). +- Disconnect-recreate gate (finding #2); FileBrowser unmounted-skeleton gate (finding #3). +- `/api/backup/snapshots` handler + UI snapshot selection; restore `--clean` option (campaign-#1 gaps).