264 lines
15 KiB
Markdown
264 lines
15 KiB
Markdown
# REPORT — TASK-D Parts 1–2: dead-primary alerting (R-51) + boot reconciliation (R-52)
|
||
|
||
**Controller v0.155.0 → v0.156.0.** Session 2026-07-21, on DooPlex as `kisfenyo`.
|
||
**Status: SHIPPED, deployed via the floor save, and STOP-1 RAN — both legs passed (§6).** P1 is
|
||
answered (§4). The validation also surfaced a real pre-existing gap, filed as **R-55** (§6). Nothing
|
||
below claims live behaviour that was not observed.
|
||
|
||
---
|
||
|
||
## 1. Baselines (re-confirmed at session start)
|
||
|
||
| Repo | `main` @ start | clean | now |
|
||
|---|---|---|---|
|
||
| felhom-controller | `0f9b29a` | yes | `285dd10` (v0.156.0) |
|
||
| felhom-agent | `08b55a1` | yes | `98adb72` (v0.92.1) |
|
||
| felhom.eu | `50a7ffa` | yes | docs commit (see §9) |
|
||
|
||
Image: `gitea.dooplex.hu/admin/felhom-controller:0.156.0`, pushed and **never hand-deployed** — the
|
||
operator's floor save deployed it, which banked the R-23 single-fire datapoint for free:
|
||
`update successful (0.155.0 → 0.156.0)` then `Current version 0.156.0 is up to date`. 9201 runs
|
||
**0.156.0**.
|
||
|
||
---
|
||
|
||
## 2. The defect R-51 actually fixes (the ROADMAP row was wrong)
|
||
|
||
The row says aggregation classifies a dead-primary stack `unhealthy`, and that `IsDownState`'s
|
||
deliberate `unhealthy` exclusion is therefore what suppresses the alarm. **That is not what the code
|
||
did.** `aggregateState`'s final branch was:
|
||
|
||
```go
|
||
// Mix (some running, some stopped) — report as running (partial)
|
||
if running > 0 { return StateRunning }
|
||
```
|
||
|
||
So the immich stack read **running**, not unhealthy. `IsDownState` was never consulted about
|
||
`unhealthy` at all, and the constraint the row protects was never in tension with the fix. The
|
||
correction is recorded in the ROADMAP row itself.
|
||
|
||
**The fix.** New `StateDegraded`. The mixed branch asks each DOWN member for its docker restart
|
||
policy: `always` / `unless-stopped` → docker was supposed to be keeping it up → the stack is
|
||
**degraded**; `no` / `on-failure` → a finished one-shot init/migrate container → benign, stack stays
|
||
running. `IsDownState` gains `degraded` **and nothing else** — `downstate_test.go` is untouched and
|
||
green, which is the fix-3 contract.
|
||
|
||
**An unreadable policy counts as supervised — fail-CLOSED**, deliberately the opposite of
|
||
`IsDownState`'s documented fail-open. They are different questions: there the *state* is ambiguous,
|
||
here a member is known dead and only its excuse is missing. P2 backs it (§4).
|
||
|
||
**Blast radius, decided per call site** (`StateDegraded` is a stack-level aggregate and never a
|
||
container state, so container-level code is untouched by construction):
|
||
|
||
| Question the caller asks | Sites | Degraded counts as |
|
||
|---|---|---|
|
||
| "are there live containers to stop?" | quiesce `RunningAppStacks`, delete stop-first guard ×2, export `IsStackRunning`, telemetry, health-probe eligibility | **up** |
|
||
| "is this app working?" | dashboard counter, stopped filter, dead-app banner + `app_start_failed` | **down** |
|
||
| "did this come back healthy?" | `waitHealthy` (migration), post-restore `RefreshAndIsRunning`, integrations readiness | **not healthy** (left strict) |
|
||
|
||
The health probe can no longer mask it: the probe result only ever overrides `StateRunning`, so a
|
||
failing probe on a degraded stack cannot downgrade it to `unhealthy` (which would silence the alarm
|
||
again). Asserted by the wiring test's second refresh.
|
||
|
||
---
|
||
|
||
## 3. R-52 — the recovery half
|
||
|
||
`internal/bootrecon`, one bounded sweep 5 s after startup (after the quiesce recovery, so the two
|
||
never race for the same stack): at most **2 attempts, 30 s apart**, then it stops and the alarm owns
|
||
the problem. Never a loop. `compose up -d` exits 0 on a crash-loop, so success is decided by a fresh
|
||
`RefreshStatus`, never by a nil error.
|
||
|
||
**The safety argument is the container gate.** The UI's Stop is `compose down`, which REMOVES the
|
||
containers; a boot interruption leaves them behind as `Exited`. So "deployed, has containers, and
|
||
they are down" is the boot-orphan signature, and a zero-container stack is never touched.
|
||
|
||
Worst case 5 s + 30 s = 35 s < the 90 s `deadAppBootGrace`, so a successful recovery never alerts
|
||
and a failure alerts honestly. `TestBootReconcileFitsInsideTheBootGrace` asserts that arithmetic
|
||
instead of leaving it in a comment.
|
||
|
||
---
|
||
|
||
## 4. Phase-0 probes
|
||
|
||
**P2 (R-51 one-shot census) — DONE, and the answer is clean.** All **53** catalog templates,
|
||
**78** services: **every single one is `unless-stopped`**. Zero services with `no` / `on-failure` /
|
||
absent. (The other 30 compose files under `output/` are generated artifacts, not templates.) So the
|
||
restart-policy guard protects a case that does not exist in today's catalog — it is there for future
|
||
templates and for the fail-closed decision above, and it costs one cached `docker inspect` per down
|
||
member of a mixed stack.
|
||
|
||
**P1 (why `unless-stopped` did not resurrect immich/calibre-web) — ANSWERED during STOP-1, and the
|
||
F5 hypothesis is CONFIRMED.** It was never made a blocker (the reconciliation is correct either
|
||
way), and it fell out of the R-52 leg for free, as predicted. bookstack carries
|
||
`restart=unless-stopped`; the Docker daemon came up at ~10:53:15Z; the container's `StartedAt` is
|
||
**`10:53:28.05Z`, the exact moment `bootrecon`'s `StartStack` returned**. Docker did not bring it
|
||
back — a container stopped before shutdown is recorded user-stopped and stays down. Details in §6.
|
||
|
||
**P3/P4** were agent-side — see `felhom-agent/REPORT.md`.
|
||
|
||
---
|
||
|
||
## 5. Tests and red-proofs
|
||
|
||
Green gate: `go build ./... && go vet ./... && go test ./...` — **all 25 packages ok** (full run,
|
||
`-count=1`; `internal/backup` alone takes 174 s).
|
||
|
||
New: `internal/stacks/degraded_test.go` (8 cases incl. two production-path tests),
|
||
`internal/bootrecon/bootrecon_test.go` (8 cases), `cmd/controller/bootrecon_wiring_test.go` (4).
|
||
|
||
| # | Red-proof | Mutation | Result |
|
||
|---|---|---|---|
|
||
| A | dead primary must alert | mix branch → `return StateRunning` | **FAIL ×3** — `aggregateState = "running", want "degraded"`; both production-path tests report `immich state = "running"`. Restored, green. |
|
||
| C | the boot sweep must be wired | `go runBootReconcile(...)` commented out in `main()` | **FAIL** — `func main() no longer starts the R-52 boot reconciliation`. Restored, green. |
|
||
| D | a user-stopped app must stay stopped | zero-container gate dropped from `isBootOrphan` | **FAIL ×2** — `StartStack("jellyfin") called 1 times, want 0 — a deliberate Stop must survive a reboot`. Restored, green. |
|
||
|
||
**A finding from red-proof C, worth more than the fix it guards.** That wiring test was first
|
||
written as `strings.Contains(src, "go runBootReconcile(ctx, stackMgr, logger)")`. Its red-proof
|
||
**passed** — because commenting the call out leaves the string in the file. A test whose own
|
||
red-proof cannot fail it is decoration. Rewritten as an AST walk over `func main()`'s body for a
|
||
`GoStmt` calling `runBootReconcile`; the red-proof then failed as it must. Recorded in REUSE.md and
|
||
promoted into `documentation/PROMPT-TEMPLATE.md` §10 (Part-5 rider).
|
||
|
||
**Seam discipline (§9 rule 6).** Both features have a production-path test:
|
||
- R-51: `TestRefreshStatus_WiresDegradedThroughTheRealPath` drives the whole real chain —
|
||
`RefreshStatus` → `docker ps` → `aggregateState` → `docker inspect` — through a new `execFn` seam
|
||
at the process boundary, and asserts the inspect COUNT (exactly 1: only the down member of the
|
||
mixed stack, never the healthy stack; still 1 after a second refresh, proving the cache).
|
||
- R-52: the sweep is asserted from `package main`, plus the AST wiring test above.
|
||
|
||
---
|
||
|
||
## 6. Live validation — STOP-1 RAN, both legs PASSED (operator-present)
|
||
|
||
Method: the exact endpoints the UI calls, driven from inside guest 9201 through traefik
|
||
(`--resolve felhom.demo-felhom.eu:443:127.0.0.1`, session cookie + CSRF token), plus container-level
|
||
truth from `docker ps -a` and the controller's own log. No browser (none on DooPlex). The session
|
||
cookie jar was deleted from the guest afterwards.
|
||
|
||
**Deploy first, per the resolved ordering:** the operator saved the floor → `0.156.0`, the box
|
||
self-updated, and the R-23 datapoint came free —
|
||
`[selfupdate] Post-update startup: update successful (0.155.0 → 0.156.0)` then
|
||
`Current version 0.156.0 is up to date`: **one swap, no re-fire.**
|
||
|
||
### Leg A — R-51, the dead primary (12:50:40 → 12:51:52 CEST)
|
||
|
||
| Time | Event |
|
||
|---|---|
|
||
| 12:50:40 | `docker stop immich-server` → `exited | restart-policy=unless-stopped`; the three helpers stay `running` — the exact F4 shape |
|
||
| **12:50:53** | **stack state = `degraded`** (13 s). `immich-server -> exited | Exited (143)`, three helpers `running`. This read `running` for 18 hours on 2026-07-20 |
|
||
| 10:51:11Z | **exactly ONE** `Event pushed: app_start_failed (warn) — Telepített alkalmazás nem fut: Immich` — single-fire verified by `grep -c`, = 1 |
|
||
| — | Dashboard: banner *"Telepített alkalmazás nem fut: Immich (degraded)"* + link *Rendszermonitor*, and the new „Részlegesen leállt" label present in the rendered page |
|
||
| 12:51:37 | `docker start immich-server` |
|
||
| 12:51:52 | state back to `running`; **banner self-cleared** (state-based, as designed) |
|
||
|
||
The `(degraded)` in the banner is the English-state wart predicted in §7 — visible now, pre-existing,
|
||
still a copy decision rather than a bug.
|
||
|
||
### Leg B — R-52, the boot orphans (12:53:07 CEST reboot)
|
||
|
||
Fixture: `docker stop` on **bookstack** (2 containers) and **calibre-web** (containers left in
|
||
place); **UI Stop** on **immich** via `POST /api/stacks/immich/stop` (`compose down` → 0 containers).
|
||
Then `pct reboot 9201`.
|
||
|
||
```
|
||
10:53:22Z [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [bookstack] — up to 2 attempt(s)
|
||
10:53:28Z [bootrecon] attempt 1/2: started "bookstack" (took 6.0s)
|
||
10:53:28Z [bootrecon] complete: 1 app(s) recovered in 1 attempt(s): [bookstack]
|
||
```
|
||
|
||
**Zero `app_start_failed`** since the reboot — a successful recovery inside the boot grace is silent,
|
||
which was the design claim. And the quiet path is observable on the healthy box too: the earlier
|
||
0.156.0 boot logged `Boot reconciliation: no boot-orphaned apps (nothing to start)`, so "nothing to
|
||
do" and "never ran" are distinguishable in production, not just in a test.
|
||
|
||
### P1 — answered, and the F5 hypothesis CONFIRMED
|
||
|
||
bookstack carries `restart=unless-stopped`. The Docker daemon came up at ~10:53:15Z. The container's
|
||
`StartedAt` is **`10:53:28.05Z` — the exact moment `bootrecon`'s `StartStack` returned.** Docker's
|
||
own restart policy did **not** resurrect it: a container stopped before shutdown is recorded
|
||
user-stopped and stays down across the reboot. Only R-52 brought it back. That is the F5 hypothesis
|
||
the audit could not test, settled with direct evidence and at zero extra cost.
|
||
|
||
### The finding this leg produced — new R-55 (it is the opposite of what the leg set out to prove)
|
||
|
||
Leg B was designed to show that a customer's deliberate Stop survives a reboot. **It does not — for
|
||
any drive-backed app.** immich, stopped from the UI seconds earlier, came back **running**:
|
||
|
||
```
|
||
10:53:18Z [gate] boot …: live bind confirmed — recreating drive-backed app immich (state=stopped) onto /mnt/felhom-drives/hdd_1
|
||
10:53:18Z [stacks] Stopping stack: immich → Starting stack: immich
|
||
```
|
||
|
||
`internal/web/intermediary.go`'s boot bind gate selects on `cfg.Deployed && HDD_PATH` alone and runs
|
||
`StopStack` + `StartStack` on every match; the stack's `state` is passed in **for the log line only**
|
||
and never consulted. calibre-web went the same way; bookstack, not drive-backed, fell through to
|
||
R-52 — which is why the sweep found exactly one orphan.
|
||
|
||
**R-52 is not implicated.** Its own gate behaved exactly as specified: immich, at zero containers,
|
||
was never a candidate. But two things follow and need an operator ruling rather than a drive-by fix
|
||
(the gate exists to fix a real bind-ordering bug and must keep doing that):
|
||
|
||
1. A customer's Stop is silently undone on drive-backed apps at every reboot.
|
||
2. **R-52's practical scope on a real box is narrower than the brief assumed** — the gate reaches
|
||
every drive-backed app first, so the boot-orphan sweep only ever sees the remainder.
|
||
|
||
Filed as **R-55**; likely shape is that the gate should recreate only apps that were RUNNING at
|
||
shutdown, i.e. the same running/stopped distinction R-52 already draws.
|
||
|
||
## 6b. What was NOT claimed live
|
||
|
||
**Nothing beyond the above.** What is verified without the box: the full green gate, all four design gates
|
||
(`template_id_gate.py`, `emoji_gate.py`, `native_confirm_gate.py`, `offbox_rename_gate.py` — all
|
||
OK), and the image build+push.
|
||
|
||
---
|
||
|
||
## 7. UI surfaces (Hungarian)
|
||
|
||
New state label „**Részlegesen leállt**", `warn` colour token, `◑` icon (shared with unhealthy),
|
||
counted with the stopped apps, filtered into the stopped bucket, and `routeUnpublished` → true (when
|
||
the dead member is the routed one Traefik withholds the route and the URL 404s — the card must not
|
||
imply the app is reachable). **No new customer strings were needed for the alarm itself**: the
|
||
existing `app_start_failed` sentence („Telepített alkalmazás nem fut: …") is reused unchanged, which
|
||
is why R-51 ships without touching the notifier.
|
||
|
||
One honest wart carried forward, not introduced: the dead-app banner appends the raw state in
|
||
English — it will now read `(degraded)` where it read `(stopped)`. Pre-existing behaviour; changing
|
||
it is a copy decision, not part of this fix.
|
||
|
||
---
|
||
|
||
## 8. A workspace trap found the hard way (worth more than this task)
|
||
|
||
`controller/.gitignore` line 7 is `controller` — intended for the built binary. It also matches the
|
||
**directory** `controller/cmd/controller/`. Two consequences, both live today:
|
||
|
||
1. **ripgrep (and therefore the Grep tool) silently skips `cmd/controller/main.go`** — the entire
|
||
production wiring file. Early in this session that produced a false "the whole fix-3 dead-app
|
||
path has no production caller" reading — exactly the inert-seam conclusion this task is about.
|
||
`grep -rn --no-ignore` / plain `grep -rn` (without `--include`) shows it.
|
||
2. **New files under `cmd/controller/` need `git add -f`.** `bootrecon_wiring_test.go` was refused
|
||
by a plain `git add` — as part of a multi-path add, i.e. quietly. A wiring test that never
|
||
reaches the repo is the same defect class it exists to prevent.
|
||
|
||
Recorded in the commit message and REUSE.md. Worth a `.gitignore` fix (anchored `/controller`) as
|
||
its own XS change — deliberately not smuggled into this task.
|
||
|
||
---
|
||
|
||
## 9. Deliverables
|
||
|
||
- `285dd10` — v0.156.0 code + tests + CHANGELOG + REUSE + README.
|
||
- Image `felhom-controller:0.156.0` pushed; **not deployed**.
|
||
- Docs in `felhom.eu`: ROADMAP R-51/R-52 → SHIPPED (with the corrected R-51 diagnosis), new R-54
|
||
row, a new capability-map row (IMPLEMENTED — not PROVEN-LIVE, the live legs have not run), and the
|
||
two Part-5 riders.
|
||
|
||
## 10. Operator actions outstanding
|
||
|
||
1. **STOP-1** (controller legs) — needs 0.156.0 live; see the ordering question in
|
||
`felhom.eu/REPORT.md` §5.
|
||
2. **STOP-3** floor → `0.156.0`, which is also what deploys it.
|
||
3. P1's answer falls out of STOP-1's reboot leg for free — worth recording either way.
|