felhom-controller/REPORT.md

REPORT — TASK-D Parts 1–2: dead-primary alerting (R-51) + boot reconciliation (R-52)

Controller v0.155.0 → v0.156.0. Session 2026-07-21, on DooPlex as kisfenyo. Status: code SHIPPED and image published; the three live STOP legs are operator-present and have NOT run. Nothing below claims live behaviour that was not observed.

---

1. Baselines (re-confirmed at session start)

Repo	main @ start	clean	now
felhom-controller	0f9b29a	yes	285dd10 (v0.156.0)
felhom-agent	08b55a1	yes	98adb72 (v0.92.1)
felhom.eu	50a7ffa	yes	docs commit (see §9)

Image: gitea.dooplex.hu/admin/felhom-controller:0.156.0, pushed, deliberately not hand-deployed — the floor save at STOP-3 deploys it, which banks another single-fire self-update datapoint free (the R-23 pattern). 9201 still runs 0.155.0.

---

2. The defect R-51 actually fixes (the ROADMAP row was wrong)

The row says aggregation classifies a dead-primary stack unhealthy, and that IsDownState's deliberate unhealthy exclusion is therefore what suppresses the alarm. That is not what the code did. aggregateState's final branch was:

// Mix (some running, some stopped) — report as running (partial)
if running > 0 { return StateRunning }

So the immich stack read running, not unhealthy. IsDownState was never consulted about unhealthy at all, and the constraint the row protects was never in tension with the fix. The correction is recorded in the ROADMAP row itself.

The fix. New StateDegraded. The mixed branch asks each DOWN member for its docker restart policy: always / unless-stopped → docker was supposed to be keeping it up → the stack is degraded; no / on-failure → a finished one-shot init/migrate container → benign, stack stays running. IsDownState gains degraded and nothing else — downstate_test.go is untouched and green, which is the fix-3 contract.

An unreadable policy counts as supervised — fail-CLOSED, deliberately the opposite of IsDownState's documented fail-open. They are different questions: there the state is ambiguous, here a member is known dead and only its excuse is missing. P2 backs it (§4).

Blast radius, decided per call site (StateDegraded is a stack-level aggregate and never a container state, so container-level code is untouched by construction):

Question the caller asks	Sites	Degraded counts as
"are there live containers to stop?"	quiesce RunningAppStacks, delete stop-first guard ×2, export IsStackRunning, telemetry, health-probe eligibility	up
"is this app working?"	dashboard counter, stopped filter, dead-app banner + app_start_failed	down
"did this come back healthy?"	waitHealthy (migration), post-restore RefreshAndIsRunning, integrations readiness	not healthy (left strict)

The health probe can no longer mask it: the probe result only ever overrides StateRunning, so a failing probe on a degraded stack cannot downgrade it to unhealthy (which would silence the alarm again). Asserted by the wiring test's second refresh.

---

3. R-52 — the recovery half

internal/bootrecon, one bounded sweep 5 s after startup (after the quiesce recovery, so the two never race for the same stack): at most 2 attempts, 30 s apart, then it stops and the alarm owns the problem. Never a loop. compose up -d exits 0 on a crash-loop, so success is decided by a fresh RefreshStatus, never by a nil error.

The safety argument is the container gate. The UI's Stop is compose down, which REMOVES the containers; a boot interruption leaves them behind as Exited. So "deployed, has containers, and they are down" is the boot-orphan signature, and a zero-container stack is never touched.

Worst case 5 s + 30 s = 35 s < the 90 s deadAppBootGrace, so a successful recovery never alerts and a failure alerts honestly. TestBootReconcileFitsInsideTheBootGrace asserts that arithmetic instead of leaving it in a comment.

---

4. Phase-0 probes

P2 (R-51 one-shot census) — DONE, and the answer is clean. All 53 catalog templates, 78 services: every single one is unless-stopped. Zero services with no / on-failure / absent. (The other 30 compose files under output/ are generated artifacts, not templates.) So the restart-policy guard protects a case that does not exist in today's catalog — it is there for future templates and for the fail-closed decision above, and it costs one cached docker inspect per down member of a mixed stack.

P1 (why unless-stopped did not resurrect immich/calibre-web) — NOT RUN, and deliberately not a blocker. It requires stopping a container on a live guest and rebooting it: STOP-1's territory, operator-present. The reconciliation is correct either way — if Docker did record them as user-stopped, unless-stopped will never bring them back and only R-52 can; if it did not, R-52 is a harmless no-op because the containers are already up when the sweep looks. Recorded as an open question, not a dependency. It can be answered for free during STOP-1, whose R-52 leg reboots 9201 with containers stopped — precisely P1's experiment.

P3/P4 were agent-side — see felhom-agent/REPORT.md.

---

5. Tests and red-proofs

Green gate: go build ./... && go vet ./... && go test ./... — all 25 packages ok (full run, -count=1; internal/backup alone takes 174 s).

New: internal/stacks/degraded_test.go (8 cases incl. two production-path tests), internal/bootrecon/bootrecon_test.go (8 cases), cmd/controller/bootrecon_wiring_test.go (4).

#	Red-proof	Mutation	Result
A	dead primary must alert	mix branch → return StateRunning	FAIL ×3 — aggregateState = "running", want "degraded"; both production-path tests report immich state = "running". Restored, green.
C	the boot sweep must be wired	go runBootReconcile(...) commented out in main()	FAIL — func main() no longer starts the R-52 boot reconciliation. Restored, green.
D	a user-stopped app must stay stopped	zero-container gate dropped from isBootOrphan	FAIL ×2 — StartStack("jellyfin") called 1 times, want 0 — a deliberate Stop must survive a reboot. Restored, green.

A finding from red-proof C, worth more than the fix it guards. That wiring test was first written as strings.Contains(src, "go runBootReconcile(ctx, stackMgr, logger)"). Its red-proof passed — because commenting the call out leaves the string in the file. A test whose own red-proof cannot fail it is decoration. Rewritten as an AST walk over func main()'s body for a GoStmt calling runBootReconcile; the red-proof then failed as it must. Recorded in REUSE.md and promoted into documentation/PROMPT-TEMPLATE.md §10 (Part-5 rider).

Seam discipline (§9 rule 6). Both features have a production-path test:

• R-51: TestRefreshStatus_WiresDegradedThroughTheRealPath drives the whole real chain — RefreshStatus → docker ps → aggregateState → docker inspect — through a new execFn seam at the process boundary, and asserts the inspect COUNT (exactly 1: only the down member of the mixed stack, never the healthy stack; still 1 after a second refresh, proving the cache).
• R-52: the sweep is asserted from package main, plus the AST wiring test above.

---

6. Live validation

None claimed for R-51/R-52. The three legs are destructive and operator-present. The available method here is endpoint-level (no browser on DooPlex); these legs additionally need a container kill and a guest reboot on the production demo box, so they are STOP-1, not a CC action.

What IS verified without the box: the full green gate, all four design gates (template_id_gate.py, emoji_gate.py, native_confirm_gate.py, offbox_rename_gate.py — all OK), and the image build+push.

---

7. UI surfaces (Hungarian)

New state label „Részlegesen leállt", warn colour token, ◑ icon (shared with unhealthy), counted with the stopped apps, filtered into the stopped bucket, and routeUnpublished → true (when the dead member is the routed one Traefik withholds the route and the URL 404s — the card must not imply the app is reachable). No new customer strings were needed for the alarm itself: the existing app_start_failed sentence („Telepített alkalmazás nem fut: …") is reused unchanged, which is why R-51 ships without touching the notifier.

One honest wart carried forward, not introduced: the dead-app banner appends the raw state in English — it will now read (degraded) where it read (stopped). Pre-existing behaviour; changing it is a copy decision, not part of this fix.

---

8. A workspace trap found the hard way (worth more than this task)

controller/.gitignore line 7 is controller — intended for the built binary. It also matches the directory controller/cmd/controller/. Two consequences, both live today:

1. ripgrep (and therefore the Grep tool) silently skips cmd/controller/main.go — the entire production wiring file. Early in this session that produced a false "the whole fix-3 dead-app path has no production caller" reading — exactly the inert-seam conclusion this task is about. grep -rn --no-ignore / plain grep -rn (without --include) shows it.
2. New files under cmd/controller/ need git add -f. bootrecon_wiring_test.go was refused by a plain git add — as part of a multi-path add, i.e. quietly. A wiring test that never reaches the repo is the same defect class it exists to prevent.

Recorded in the commit message and REUSE.md. Worth a .gitignore fix (anchored /controller) as its own XS change — deliberately not smuggled into this task.

---

9. Deliverables

• 285dd10 — v0.156.0 code + tests + CHANGELOG + REUSE + README.
• Image felhom-controller:0.156.0 pushed; not deployed.
• Docs in felhom.eu: ROADMAP R-51/R-52 → SHIPPED (with the corrected R-51 diagnosis), new R-54 row, a new capability-map row (IMPLEMENTED — not PROVEN-LIVE, the live legs have not run), and the two Part-5 riders.

10. Operator actions outstanding

1. STOP-1 (controller legs) — needs 0.156.0 live; see the ordering question in felhom.eu/REPORT.md §5.
2. STOP-3 floor → 0.156.0, which is also what deploys it.
3. P1's answer falls out of STOP-1's reboot leg for free — worth recording either way.