admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: SPIKE R-50 island-bridge — drill env ABSENT (empirical blocked); source design findings + dnsmasq trap; ROADMAP R-50 spiked

Browse Source
This commit is contained in:
authentik Default Admin
2026-07-25 08:30:25 +02:00
parent be2d90f243
commit 1d65f44205
2 changed files with 119 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/SPIKE-island-bridge-2026-07-25.md
+118
View File
@@ -0,0 +1,118 @@
# SPIKE — R-50 island-bridge control plane, 2026-07-25
**Question:** the agent binds a LAN literal (`local_api.listen_addr`) and the guest dials that literal
from `bootstrap.json`. A DHCP/site change that moves the host's LAN address makes the agent **fail to
start** (`bind: cannot assign requested address`) — storage/PBS/quiesce/restore-test/DR all go down
silently (the 2026-07-20 vacation incident, F1). Target: a host-internal bridge (no physical port) with
fixed private addresses, so the control plane survives any router/lease/site move. This spike was to
**validate the whole chain end-to-end on the DRILL environment (qm300 / `demo-vm-felhom-2f4b00`)**
before any production spec exists.
## VERDICT UP FRONT: SPIKE BLOCKED (empirically) — drill environment is GONE.
**Probe 1 (drill env health check) — FAIL/ABSENT.** On felhom-pve (2026-07-25):
```
# qm list → (empty) — no QEMU VMs at all; the nested drill PVE VM qm300 does not exist
# pct list → VMID 9201 "demo-felhom" running — the LIVE demo controller only
# grep -rl '2f4b00|drill' /etc/pve/{qemu-server,lxc}/ → (no matches)
```
The drill appliance (a throwaway nested PVE-in-a-VM) has been torn down since the last drill. **Per the
task's hard environment rule — "if the drill env is down or unreachable: STOP, report, done; do not
improvise on production" — probes 3–8 (bridge create, guest NIC hot-add, island bind, the F1 replay,
survival matrix) were NOT run.** felhom-pve's bridges/NICs, the live agent's `listen_addr`, and guest
9201 were left **UNTOUCHED**. **No GO/NO-GO can be issued on live evidence; a rebuilt drill VM is
required to validate the chain.**
What follows is the **source-only analysis** (read-only; no host mutation) — the address plan, the
three named findings, and the provisioning inventory — so the eventual implementation spec (and the
next drill run) start from a real base rather than zero.
---
## Probes 3–8 — NOT RUN (drill env absent)
| Probe | Would validate | Status |
|---|---|---|
| 3 Bridge create (`vmbr9`, no ports, `ifreload -a`) w/ vmbr0 untouched | non-disruptive host bridge add | **BLOCKED** — needs drill host |
| 4 Guest NIC hot-add (`pct set -net1 …ip=169.254.253.2/30`) | live hotplug, LAN leg undisturbed | **BLOCKED** |
| 5 Agent island bind + **F1 replay** (renumber LAN → agent still starts/serves) | the money shot | **BLOCKED** |
| 6 Pin confirmation (bootstrap→island, controller round-trip, unchanged fingerprint) | pin survives address move | **BLOCKED** (but see source finding below) |
| 7 dnsmasq trap probe | LAN DNS on the LAN bridge, not the island | **BLOCKED** (source-confirmed below) |
| 8 Survival matrix (agent restart / guest reboot / host reboot) | control plane returns on the island | **BLOCKED** |
---
## Source-confirmed findings (read-only; no probe needed)
### F1 root cause — the LAN literal is baked in TWO places, together
- **Agent bind:** `felhom-agent/internal/config/config.go:229` — `ListenAddr string` "bridge IP:port,
e.g. `192.168.0.162:8443`"; validated as `host:port` (`:280–284`). The install script sets it from
the **vmbr0 (LAN) IP**: `felhom.eu/scripts/felhom-host-install.sh:2162` (`ip -4 addr show vmbr0`) →
`:2226` (`base['local_api']['listen_addr'] = BRIDGE_ADDR`), `--bridge-ip` default = "vmbr0 IP : 8443"
(`:114`).
- **Guest dial:** `felhom-agent/internal/provision/backhalf.go:75/129` writes that same `Endpoint`
(bridge IP:port) into `bootstrap.json`'s `local_api.endpoint` (`internal/provision/doc.go:44`) — the
address the in-guest controller dials.
→ Both must move to the island address **atomically** (agent bind + the guest's bootstrap endpoint), or
the controller can't reach the agent.
### Finding 1 — the dnsmasq trap (CONFIRMED in source; a MUST for the spec)
`felhom-agent/internal/config/config.go:182–210`: `LANResolverConfig.HostIP` is the **dnsmasq
listen-address**, and `WithDefaults(localAPIListen)` **derives it from the local-API bind addr** when
unset (`:208–210`, `l.HostIP = host(localAPIListen)`). So the instant `listen_addr` moves to the island
(`169.254.253.1`), the split-horizon LAN resolver would bind its DNS to the **island** bridge and **LAN
DNS dies** — a silent, nasty coupling. **Spec requirement:** set `LANResolverConfig.HostIP` **explicitly
to the LAN (vmbr0) bridge IP**; never let it default off `listen_addr` once the two diverge. (The
install script must write `lan_resolver.host_ip` alongside the new `listen_addr`.)
### Finding 6 (pin) — moving the bind should NOT need a cert re-issue
The controller pins the agent's **leaf-cert DER SHA-256**, not its SAN/address:
`felhom-agent/internal/localapi/cert_test.go:37` ("the reported fingerprint must equal the SHA-256 of
the served leaf DER"); the pinning verifier keys on the cert bytes, not the connect address. So an
address move is transparent to the pin **as long as the same leaf cert is served** — no re-issue
expected. *(Unproven live — that was probe 6; asserted from source.)* Caveat: if a future cert is minted
with the island IP in its SAN and something ever DOES check SAN, that changes — but today's pin does not.
### Address-plan recommendation
- **Recommend link-local `169.254.253.1/30` (host) ↔ `169.254.253.2/30` (guest) on `vmbr9`.**
Rationale: RFC-3927 link-local (`169.254.0.0/16`) is, by standard, **neither routed nor DHCP-assigned
on a LAN** — a home router won't hand it out or forward it, so a host-internal /30 cannot collide with
any customer LAN, ever. A /30 is exactly two usable hosts (host + guest) — no waste, no ambiguity.
- Alternative (obscure RFC-1918 /30, e.g. `10.201.253.0/30`): also works, but a customer LAN *could*
legitimately use `10.0.0.0/8` and route it, so it carries a small non-zero collision risk that
link-local structurally does not. **Prefer link-local.**
### Provisioning inventory (what an implementation must touch — file:line)
| Concern | Location |
|---|---|
| Create `vmbr9` (portless) + host island IP | `felhom.eu/scripts/felhom-host-install.sh` (new: `/etc/network/interfaces` stanza + `ifreload -a`) |
| Guest `net1` on `vmbr9` (static island IP) | `felhom-host-install.sh` (new `pct set … -net1 …`) — **R-54 interplay:** guest **eth0 stays DHCP** (the guestnet healer keeps owning the LAN leg); the island NIC is static; the control plane simply stops depending on the LAN leg |
| `listen_addr` → island IP | `felhom-host-install.sh:2226` (+ default logic `:2162/:114`) → agent `config.go:229` |
| `lan_resolver.host_ip` → **LAN** vmbr0 IP (Finding 1) | new write in `felhom-host-install.sh`; consumed by `config.go:198 WithDefaults` |
| `bootstrap.json` endpoint → island IP | `felhom-agent/internal/provision/backhalf.go:129` (`Endpoint`) + the golden/bootstrap template |
| Leaf cert | no change expected (pin is DER-based; Finding 6) |
### Cluster-parity finding (Peti's two-node cluster)
A one-host demo needs `vmbr9` on that host only. A **cluster** needs **bridge parity on BOTH nodes** —
either per-node identical `/etc/network/interfaces` stanzas (simplest, drift-prone) or a Proxmox **SDN**
zone/vnet defined cluster-wide (one definition, auto-applied per node; preferred at ≥2 nodes). The
guest's island IP is per-guest and node-independent; the **agent-follows-guest** rule (each host's agent
binds its own `vmbr9` island IP) holds on both nodes. Migration order for the eventual rollout:
**drill-proven → demo (one host) → Peti (SDN on the 2-node cluster)**.
---
## Implementation-spec skeleton (for the future task — NOT this one)
1. **Phase A (agent + config):** teach the install/provision path to create `vmbr9`, add guest `net1`,
set `listen_addr` + `bootstrap.json` endpoint to the island IP, and write `lan_resolver.host_ip = LAN
IP` (Finding 1). Repos: `felhom.eu` (host-install), `felhom-agent` (provision/bootstrap template).
**MinAgent/floor:** additive to provisioning; no controller coupling → **MinAgent unchanged**; a
golden re-bake carries the new bootstrap template.
2. **Phase B (migration):** per-box runbook (bridge add → net1 → agent+bootstrap renumber → restart →
F1-replay check → rollback = revert `listen_addr`+endpoint to the LAN IP), drill-first.
3. **Phase C (cluster):** SDN vnet on Peti's cluster; parity check on both nodes.
**STOP — the implementation is its own future task. This spike is BLOCKED on a rebuilt drill VM for the
empirical (probes 3–8) half; the design half above is source-grounded and ready.**
documentation/backlog/ROADMAP.md
+1 -1
View File
@@ -58,7 +58,7 @@
| R-35 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. | S | idea | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` |
| R-36 | **Post-RESET re-enroll leaves offsite "enabled but unprovisioned" — silently.** The hub knows the state and says nothing on the customer page. | S | **SHIPPED (hub v0.67.0, 2026-07-18)** | Both halves delivered. **(1) The warning:** the customer page now names the state and the fix — enabled-but-unprovisioned raises an amber banner saying provisioning is *Save*-triggered (press Save once, then verify), reusing the exact `enabled && type == ""` predicate the offsite re-issue handler already refuses on. **(2) The related sub-item, also done:** the self-bind link is now **auto-minted at customer creation AND at RESET completion**, so the console banner's „e-mailben kapott link" is already true instead of true-once-the-operator-remembers. Extracting the shared `mintAndSendSelfBindLink` core keeps the button and the auto-mint callers on the same F1/F2 honesty rules, and the auto-mint never fails the operation it rides on. **Gap found and closed while wiring it:** `PurgeCustomerResetDBState` does NOT clear `selfbind_tokens`, so a link minted BEFORE a reset would have stayed live across it — the skip paths now clear stale tokens, giving the invariant "after auto-mint, the only live link is one we just issued, or none". Tests assert the banner is ABSENT in all three nominal cases too; red-proofed. — Original analysis: Source-cited behaviour, confirmed live in the rehearsal: **provisioning is Save-triggered** (`configs.go` `applyOffsite`) — which also answers S6's open question — and the re-enroll auto-re-issue **correctly** skips unprovisioned targets (`handler.go`). So nothing is broken; the gap is that nobody is told. Direction: flash it on the customer page. **Interim: an R-3 step.** **Related sub-item:** auto-mint the **self-bind link on customer create/RESET**, so the console banner's „e-mailben kapott link" is always already true instead of true-once-the-operator-remembers |
| R-27c | **Customer self-bind, slice 2 — console-passphrase bind.** Viktor's direction: bind using a passphrase shown on the box console, alongside (not instead of) the emailed capability link. | M | idea | **Security constraints from the session ruling, all load-bearing:** passphrase **issued at customer creation**; the global-lookup endpoint must be **spray-hardened** — per-appliance **and** per-IP caps, constant-time comparison, a **single generic failure** (no oracle), alerting on abuse; an **accent-free wordlist** (console keymaps are not Hungarian); the **web capability-link path is RETAINED**; **claim-by-email is RETAINED** as the delivery-channel proof. **Also under this item:** the self-bind email gains the **public universal-ISO download link + two-line instructions** (the DIY case). **Secret-bearing per-customer ISOs are ruled OUT.** Sibling of R-27b (second-box flow) — different axis, both build on the same `/bind/` page |
| R-50 | **[P2-HIGH] Island-bridge control plane — make controller↔agent independent of the LAN.** The agent's `localapi` binds a **LAN literal** (`listen_addr`) and the guest dials that same literal from `bootstrap.json`. Move both onto a **host-internal bridge with a fixed, private address** that no router, DHCP lease, or site move can invalidate, so the control plane is location-independent and golden-image-stable. | M | **idea — SPIKE-FIRST** | Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F1), where this failed for real. The demo box moved to a remote site, DHCP handed it `.147` instead of `.162`, and the agent then **could not start at all** — `bind: cannot assign requested address`, systemd gave up after 4 retries — taking storage, PBS backup, quiesce, restore-test and DR down for as long as nobody noticed. Mitigated for that window by pinning `vmbr0` static back to `.162`; that is a **window mitigation, not the fix** — it still depends on the site's subnet being `192.168.0.0/24` and free at that address. **Spike-first is mandatory:** validate end-to-end on the drill environment (agent bind + guest dial + TLS SAN/pin + reinstall/golden survival + the bootstrap-config migration for already-deployed boxes) BEFORE any production spec. **Pin fact (verified 2026-07-20, `agentapi/client.go` L105-129 — supersedes the earlier "the SAN set must cover the new address" note in this entry, which was wrong):** the controller-to-agent leg sets `InsecureSkipVerify: true` and replaces chain verification with a custom `VerifyPeerCertificate` that does a raw **SHA-256 match on the leaf DER** against the bootstrap fingerprint. Hostname/SAN therefore never enters verification on this leg, so moving the agent listen address most likely needs **no cert re-issuance** - only the endpoint the guest dials. The spike must still confirm this empirically rather than trust the read. Flips a future "box survives a site/network change" map row |
| R-50 | **[P2-HIGH] Island-bridge control plane — make controller↔agent independent of the LAN.** The agent's `localapi` binds a **LAN literal** (`listen_addr`) and the guest dials that same literal from `bootstrap.json`. Move both onto a **host-internal bridge with a fixed, private address** that no router, DHCP lease, or site move can invalidate, so the control plane is location-independent and golden-image-stable. | M | **SPIKED (partial) 2026-07-25 — empirical BLOCKED, design half source-grounded** | **2026-07-25 spike (`audits/SPIKE-island-bridge-2026-07-25.md`): the drill environment (qm300 / `demo-vm-felhom-2f4b00`) is GONE** (`qm list` empty; only guest 9201 remains), so per the hard drill-only rule the empirical probes (bridge create / NIC hot-add / island bind / **F1 replay** / survival) were NOT run — production was left untouched, **no live GO/NO-GO**. The **source-grounded** half IS done: recommend **link-local `169.254.253.1/30`↔`.2/30`** (structurally uncollidable vs LAN); the F1 literal is baked in TWO places to move atomically (`config.go:229` + `felhom-host-install.sh:2226` for the bind, `provision/backhalf.go:129`/`bootstrap.json` for the guest dial); **NEW dnsmasq trap CONFIRMED in source** — `LANResolverConfig.WithDefaults` (`config.go:208–210`) derives the DNS listen-addr from `listen_addr`, so moving the bind to the island silently kills LAN DNS → the spec MUST set `lan_resolver.host_ip = LAN IP` explicitly; pin is leaf-DER-based (address-independent → no cert re-issue expected); provisioning inventory + cluster-parity (SDN on Peti's 2 nodes) + an implementation skeleton recorded. **Remaining: a rebuilt drill VM to validate probes 3–8 before the production spec.** Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F1), where this failed for real. The demo box moved to a remote site, DHCP handed it `.147` instead of `.162`, and the agent then **could not start at all** — `bind: cannot assign requested address`, systemd gave up after 4 retries — taking storage, PBS backup, quiesce, restore-test and DR down for as long as nobody noticed. Mitigated for that window by pinning `vmbr0` static back to `.162`; that is a **window mitigation, not the fix** — it still depends on the site's subnet being `192.168.0.0/24` and free at that address. **Spike-first is mandatory:** validate end-to-end on the drill environment (agent bind + guest dial + TLS SAN/pin + reinstall/golden survival + the bootstrap-config migration for already-deployed boxes) BEFORE any production spec. **Pin fact (verified 2026-07-20, `agentapi/client.go` L105-129 — supersedes the earlier "the SAN set must cover the new address" note in this entry, which was wrong):** the controller-to-agent leg sets `InsecureSkipVerify: true` and replaces chain verification with a custom `VerifyPeerCertificate` that does a raw **SHA-256 match on the leaf DER** against the bootstrap fingerprint. Hostname/SAN therefore never enters verification on this leg, so moving the agent listen address most likely needs **no cert re-issuance** - only the endpoint the guest dials. The spike must still confirm this empirically rather than trust the read. Flips a future "box survives a site/network change" map row |
| R-50b | **[P2] A root-owned privileged host artifact is delivered unversioned from `main` — "which wrapper is on this host?" is unanswerable.** `configs/felhom-pbs-apply` installs to `/usr/local/sbin/felhom-pbs-apply` (0755 root:root) and is the pinned sudoers vector for `create\|reconcile\|grant` against `/etc/pve/priv/storage`. It is fetched by `felhom-host-install.sh:1914` via `fetch_raw`, which hits `raw/branch/main/<path>` — **no tag, no pin, no checksum, and no record in the Day-0 artifact manifest**, unlike the agent binary (sha256-vouched) and the golden image. Three consequences: (1) two hosts installed a week apart can carry different privileged wrapper code while both reporting the same agent version; (2) a host hotfixed in place (felhom-pve, 2026-07-18) is indistinguishable from one that fetched the same content — the fleet has no inventory of it; (3) an accidental push to `main` reaches the next install of every host with no review gate between commit and root-owned deployment. | S–M | **(a) SHIPPED 2026-07-21; (b)/(c) open** | **Surfaced 2026-07-21 while stopping the R-39 v0.90.1 publish** (`felhom-controller/REPORT.md` §5): the publish was cancelled precisely because the version number would have claimed to carry a fix that in fact rides this unversioned channel. Candidate shapes, in increasing cost: (a) record the wrapper's sha256 in the Day-0 artifact manifest beside the agent binary and have the agent report the installed file's hash, so drift is at least *visible*; (b) `fetch_raw` takes a pinned ref (tag or commit) supplied by the manifest rather than `main`; (c) the wrapper becomes a published generic-registry artifact with the same gate ladder as the agent binary. **(a) is the cheap honest first step and would have caught this class already.** Pairs with R-39 (whose remaining fleet half is specced separately) **(a) SHIPPED 2026-07-21 — hub v0.68.0 + agent v0.91.2.** `ArtifactManifest.WrapperSHA256` + an operator field; agents report the installed wrapper's sha256 each cycle and the host page surfaces a mismatch. **An unknown on EITHER side reads as quiet, never as drift** — lighting every host amber on rollout day is how a warning becomes background noise. Live confirmation of exactly the problem: felhom-pve's July-18 in-place hotfix hashed `2888f2ea…`, matching **no commit anyone could name**; it now reports `104db0a4…` against a vouchable manifest value. **(b)/(c) REMAIN OPEN:** the wrapper is still fetched unversioned from `raw/branch/main` — this makes drift *visible*, it does not fix the channel. Also recorded: the 0440 sudoers file is not agent-readable, so its drift stays invisible. |
| R-51 | **Dead-primary alerting — a multi-container app whose MAIN container is dead must alert.** Aggregation currently classifies such a stack `unhealthy`, and `IsDownState` deliberately excludes `unhealthy`, so nothing fires. | S | **SHIPPED 2026-07-21 — controller v0.156.0** | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F4). Observed live: `immich-server` was `Exited` for **18 h** with the app 100 % unreachable, and the box produced **no** dead-app banner and **no** `app_start_failed` hub event — while single-container Calibre-Web, down for the same reason, alerted correctly within 90 s. **Constraint (load-bearing): do NOT simply fold `unhealthy` into down.** That exclusion is deliberate (`stacks/manager.go` fix-3, `downstate_test.go`) and reverting it reintroduces the flapping it was added to stop. Direction: distinguish *member-container-exited* from *healthcheck-failing* in the aggregation, and treat a dead primary as down | **SHIPPED 2026-07-21 (controller v0.156.0).** **The diagnosis in this row was WRONG at the source and is corrected here:** aggregation did NOT classify the stack `unhealthy`. `aggregateState`'s final branch returned `StateRunning` for any mix of running and stopped members — the comment said "report as running (partial)" — so the stack read as RUNNING and `IsDownState` had nothing to fire on. The `unhealthy` exclusion was never involved, and the constraint it protects was therefore never in tension with the fix. New `StateDegraded`: a DOWN member whose docker restart policy is `always`/`unless-stopped` means docker was supposed to be keeping it up, so the stack is degraded (a down state, alerting through the EXISTING banner + `app_start_failed` path, unchanged); `no`/`on-failure` is a finished one-shot init/migrate container and stays benign. An UNREADABLE policy counts as supervised — fail-CLOSED, deliberately the opposite of `IsDownState`'s fail-open, because there the *state* is ambiguous while here a member is known dead and only the excuse is missing (P2 census 2026-07-21: all 53 catalog templates / 78 services are `unless-stopped`, zero one-shot containers exist today). `IsDownState` gained `degraded` and NOTHING else — the `unhealthy`/`restarting`/`paused`/`unknown` exclusions are byte-identical and `downstate_test.go` is untouched and green. Red-proof: the mix branch reverted to `return StateRunning` makes the immich fixture and both production-path tests fail with `"running"`. Live leg (STOP-1) pending. Evidence: `felhom-controller/REPORT.md` (2026-07-21) **PROVEN LIVE 2026-07-21 (STOP-1, operator-present).** On guest 9201, `docker stop immich-server` at **12:50:40 CEST** (policy `unless-stopped`, three helpers left running — the exact F4 shape). **12:50:53 — 13 seconds later — the stack read `degraded`** where it read `running` for 18 hours on 2026-07-20. `10:51:11Z` **exactly ONE** `app_start_failed (warn) — Telepített alkalmazás nem fut: Immich` (single-fire verified by count, not by eye). Dashboard rendered the banner *Telepített alkalmazás nem fut: Immich (degraded)* plus the new „Részlegesen leállt" state label. `docker start` at 12:51:37 → `running` by 12:51:52 and the banner **self-cleared** (state-based, as designed). Note the banner appends the raw state in English — `(degraded)` — which is pre-existing behaviour, not introduced here, but now more visible |
| R-52 | **Boot desired-state reconciliation — a `deployed: true` app should be running after boot.** The controller *reports* deployed-but-stopped apps (30 s `deadapp-check`) but never starts them, so an app that misses its boot start stays down until a human notices. | M | **SHIPPED 2026-07-21 — controller v0.156.0** | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F5). Observed live: the pre-transport shutdown left `immich-server` and `calibre-web` `Exited`; **10 sibling containers came back and those two did not**, and they were still down ~18 h later. **Includes root-causing why `restart: unless-stopped` did not resurrect them** — both were stopped ~25 s before power-off, so Docker most likely recorded them as user-stopped; that hypothesis is untested because the guest journal is volatile and the controller's own logs were rotated by the container recreate. Direction: a bounded start-once reconciliation (N attempts, reusing the existing boot grace), never a restart loop. Pairs with R-51 — that one is the *alarm*, this one is the *recovery* | **SHIPPED 2026-07-21 (controller v0.156.0), `internal/bootrecon`.** One bounded start-once sweep at controller startup: every deployed, non-protected, not-mid-deploy stack that still HAS containers and is down gets `StartStack`, at most **2 attempts 30 s apart**, then it stops and the alarm owns the problem. Never a restart loop. The whole sweep (5 s settle + one 30 s gap) fits inside the existing 90 s `deadAppBootGrace`, so a successful recovery never alerts and a failed one alerts honestly — asserted by a test rather than left to a comment. **The safety argument is the container gate:** the UI's Stop is `compose down`, which REMOVES the containers, while an interrupted boot leaves them behind as `Exited` — so "deployed, has containers, and they are down" is exactly the boot-orphan signature, and a zero-container stack is never touched. Red-proof: dropping that gate makes the user-stopped app get started, which is the one thing this must never do. **P1 (the `unless-stopped` root-cause probe) is deliberately NOT what this shipped on** — the reconciliation is correct whether or not Docker recorded those two containers as user-stopped, and the probe is recorded as an open question rather than a blocker. Live leg (STOP-1) pending. Evidence: `felhom-controller/REPORT.md` (2026-07-21) **PROVEN LIVE 2026-07-21 (STOP-1, operator-present), and it answered P1 for free.** Fixture on 9201: `docker stop` on bookstack (2 containers, left in place) + calibre-web, and a UI Stop on immich (`compose down` → 0 containers); then `pct reboot 9201`. Result: `10:53:22Z [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [bookstack] — up to 2 attempt(s)` → `10:53:28Z attempt 1/2: started "bookstack" (took 6.0s)` → `complete: 1 app(s) recovered in 1 attempt(s)`, and **ZERO `app_start_failed`** — a successful recovery inside the boot grace is silent, exactly as designed. **P1 IS NOW ANSWERED, and the F5 hypothesis is CONFIRMED:** bookstack carries `restart=unless-stopped`, the Docker daemon came up at ~10:53:15Z, and the container's `StartedAt` is **`10:53:28.05Z` — the exact moment `bootrecon`'s `StartStack` returned**. Docker's own restart policy did NOT resurrect it; a container stopped before shutdown is recorded user-stopped and stays down. Only R-52 brought it back. **BUT the end-to-end "a deliberate Stop survives a reboot" property is NOT true today, for a reason outside R-52 — see R-55.** R-52's own gate is correct and was observed to be: immich (0 containers) was never a candidate |