felhom.eu/documentation/tests/test-campaign-2-finding1-recovery-diagnosis.md

# Finding #1 diagnosis — does a crashed controller auto-recover? — 2026-06-22 (demo 9201)

> **Status: read-only diagnosis (no fix applied).** Resolves campaign-#2 Finding #1 ("a crashed
> felhom-controller has no automatic recovery"). **Verdict: Finding #1 is a TEST ARTIFACT.** The
> controller's `restart: unless-stopped` policy **does** auto-recover it from a genuine crash — proven
> directly. The campaign tested with `docker kill`, which Docker treats as a *manual* stop (restart
> policy intentionally suppressed); that is not a crash. No real recovery gap. Downgrade Finding #1.

## TL;DR
- Docker's **live restart-manager works** on this host under a healthy daemon: a container that exits
  **on its own** is auto-restarted (probe `--restart always`, `sleep 3` → RestartCount climbs 1→2→3→4;
  dockerd logs `restarting container manualRestart=false restartPolicy="{always 0}"`).
- **`docker kill` is treated as a manual stop** → restart policy is correctly suppressed (probe → after
  `docker kill`: `Running=false RestartCount=0`, no restart log). This is what the campaign's C1/C2 hit.
- **Definitive controller test:** `kill -9` the controller's container PID directly (a real process
  death, not the docker API) → docker **auto-restarted it: RestartCount=1, health=healthy within 5 s.**
- Daemon is **stable** (NRestarts=0, up since the C8 reboot 17:01 UTC, no OOM, 8.7 GB available) — so
  this is not a daemon-state artifact and not a genuine restart-policy bug; it's the **wrong-signal
  test**.

## Evidence

### Step 1 — daemon stability
- `docker version`: Server **29.5.3**, Go 1.26.4, containerd v2.2.4, API 1.54. (Confirmed the unusual
  "29.5.3" — a very new/edge build; relevant only as context.)
- `docker info`: overlay2, cgroup v2 (systemd), runc, **Live Restore: false**.
- `systemctl show docker`: **NRestarts=0**, ExecMainStart **2026-06-22 17:01:03 UTC** (= the C8 reboot).
  dockerd has NOT restarted since; daemon is stable.
- The campaign's **"stopping restart-manager"** lines cluster at 17:01 (post-reboot bring-up) and
  17:05–17:08 (my C6/C7 drive-disconnect/quiesce tests, which *stop* containers). That log line is
  emitted whenever a container's restart-manager is cancelled (container stop **or** daemon shutdown) —
  it is **not** proof of a daemon restart. The daemon did not flap.

### Step 2 — memory / OOM
`free -h`: 12 Gi total, **8.7 Gi available**, swap unused. No `oom`/`killed process` in dmesg/journal.
Memory pressure is **not** a factor.

### Step 3 — decisive throwaway probe (healthy daemon)
| probe | after `docker kill` |
|---|---|
| `--restart unless-stopped` | `Running=false RestartCount=0` (no restart) |
| `--restart unless-stopped` (2nd kill) | `Running=false RestartCount=0` |
| `--restart always` | `Running=false RestartCount=0` (no restart) |

Both policies fail to restart **after `docker kill`** — which first looked like a host-wide bug…

### Step 3b — natural-exit test (the disambiguator)
`docker run --restart always busybox sh -c 'sleep 3'` (exits on its own):
→ **`Running=true RestartCount=3`** (and climbing). dockerd:
`restarting container … manualRestart=false restartPolicy="{always 0}"` repeatedly.
**The restart-manager works for real exits.** So `docker kill` is being recorded as a **manual** stop
(`manualRestart` path), which Docker deliberately does not auto-restart — by design, an operator who
`stop`/`kill`s a container wants it stopped.

### Step 6 — definitive controller test (real crash, not the docker API)
`kill -9 <controller container PID 1878>` (process death, bypassing docker stop/kill):
```
before:  Running=true  RestartCount=0  policy=unless-stopped
t+5s:    Running=true  RestartCount=1  health=healthy
… stable healthy thereafter
```
**The controller auto-recovered from a real crash in ~5 s**, with the data plane (apps + tunnel)
unaffected throughout. (A 60 s timed auto-recovery guard was pre-armed and not needed; cleaned up.)

### Step 4 — the real recovery chain
- Running controller: `RestartPolicy = unless-stopped` (confirmed).
- `felhom-controller-bootstrap.service`: `Type=oneshot`, no `Restart=` — provision/boot-time bring-up
  only (correct). No other unit watches the container.
- **What recovers a crashed controller:** docker's restart policy — the **live restart-manager** on a
  crash (proven) **and** boot-time restore at daemon start (which is why campaign C8's reboot brought
  26 containers back). Two independent, working paths. No secondary fallback is needed because the
  primary (restart policy) works.

## Classification
**Finding #1 = TEST ARTIFACT.** Restart policy works; the campaign's `docker kill` is a manual stop, not
a crash, so it correctly did not auto-restart. A genuine controller crash self-heals via
`unless-stopped`. **Downgrade Finding #1** (it is not a resilience gap).

## Fix direction
- **No fix required for crash-recovery.** The controller self-heals. Do **not** add a watchdog on this
  basis — it would patch a non-problem.
- *Optional, low priority:* if defense-in-depth against an *operator/tooling* `docker stop`/`kill`
  (which intentionally won't auto-restart) is ever wanted, a tiny "controller absent ⇒ recreate"
  systemd timer could be considered — but it is **not** warranted by any observed failure.
- **Re-test hygiene (the actual lesson):** simulate a crash with a real process death
  (`kill -9` the PID, OOM, panic) — never `docker kill`/`docker stop` — when testing restart policies.

## Separate observation (not Finding #1) — boot-time drive-ordering race
At the C8 reboot (17:01:05) dockerd logged many `failed to start container … mkdir
/mnt/felhom-drives/felhom-flash/userdata: permission denied` — docker's **boot-restore started
drive-backed containers before the drives were mounted**. They recovered shortly after (the controller's
recreate-on-boot once the drives mounted; C8 ended at 26 running/0 unhealthy). Real but minor
(transient boot race); worth a follow-up to order drive-backed app start after the mount converges.
No change applied.