Controller v0.190.0. Docs only here; no hub change, no hub version bump. - audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis, including the run that produced a FALSE NEGATIVE and the mechanism behind it (the agent re-binds an unmounted drive within ~60s, so the drive gate's startup reconcile restarted the apps one second before the sweep looked). Records that the write hazard was blocked only by an ACCIDENTAL filesystem permission that no code owns and no test pins. - architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired state; the sweep observes a SETTLED fleet and each sample must refresh first; nothing is started without asking, fail-safe. Plus the durable warning: Manager.StartStack has no gate of its own. - 00-capability-map — the boot-recovery row, with the repeat count cited per N.5 (6 of 6 hard resets) rather than a bare PROVEN-LIVE. - OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW and closed the same session, marked a regression from v0.189.0. - STATUS.md — the power-cut line moved from "What's broken" to "What works right now" with its repeat count; one dated bullet in the change log. - CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is not a disproof; widening a window makes previously-unreachable overlaps reachable; and a settle detector is only as good as the freshness of what it samples — the fix's own defect, found live rather than by review.
This commit is contained in:
@@ -120,10 +120,44 @@ Applied: a lost or corrupt marker means the app is not auto-restarted **by that
|
||||
the pre-v0.189.0 position, not a new hazard. A lost `app.yaml` already means the app is not deployed.
|
||||
**Nothing here may make an absent file more dangerous than a present one.**
|
||||
|
||||
### Not migrated
|
||||
### Boot recovery reads desired, and asks before it acts (v0.190.0)
|
||||
|
||||
`shouldRecreateOnBoot` (`internal/web/intermediary.go`) — the drive-backed boot gate — still derives
|
||||
desired from observed via `hasContainers`. Tracked as **R-170**.
|
||||
**Both boot gates now read desired state.** `bootrecon.isBootOrphan` (the R-52 sweep) and
|
||||
`shouldRecreateOnBoot` (the drive-backed recreate gate) answer the same question — *did the customer
|
||||
want this running?* — with the same three-way table, absent falling back to the pre-v0.190.0
|
||||
container count in both. Their agreement is pinned from both sides against one fixture table, because
|
||||
an import cycle prevents testing them together. R-170 closed the last gate that still guessed.
|
||||
|
||||
**The sweep observes a SETTLED fleet, not a single early sample.** It samples (name, state, container
|
||||
count) every 5 s, calls the fleet settled after 3 identical samples, and sweeps **once**, at the end.
|
||||
The window ends on settled or a 50 s budget, and the log says which. Two constraints bound it:
|
||||
|
||||
- `bootReconcileSettle + budget + one DefaultRetryDelay` must stay inside `deadAppBootGrace`, or a
|
||||
successful recovery stops being silent. This is arithmetic, pinned by a test.
|
||||
- **each sample must REFRESH first.** `GetStacks()` is the Manager's in-memory map, refreshed by the
|
||||
scheduler on its own 10 s cadence; sampling it faster without refreshing lets "settled" mean "the
|
||||
cache did not update". Found by live validation, not review.
|
||||
|
||||
A recovery completing after the grace emits a `LATE RECOVERY` warning naming the apps. The grace is
|
||||
**never** widened to make a late recovery look silent.
|
||||
|
||||
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
|
||||
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
|
||||
answer it, and the last two only became reachable once the window widened past T+5 s:
|
||||
|
||||
| Holder | Why starting would be wrong |
|
||||
|---|---|
|
||||
| the drive-absent gate | compose creates bind sources wherever the mountpoint points — the guest rootfs |
|
||||
| a quiesce | a running app inside a snapshot meant to be clean-shutdown-consistent |
|
||||
| an in-flight app-data operation | restarting an app under its own tar |
|
||||
|
||||
The rule is **not new** — the API's `startGatedByMissingDrive` already refused a customer's start on
|
||||
an absent drive. The sweep bypassed it by calling `Manager.StartStack` directly, which is what R-171
|
||||
closed. Held apps are reported separately from "still down": they are not a fault the sweep failed to
|
||||
fix, and reporting them as one is a false alarm.
|
||||
|
||||
**Read this before adding a fourth caller of `Manager.StartStack`.** That method has no gate of its
|
||||
own; every caller that is not the customer must decide for itself whether the app may run.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user