Controller v0.190.0. Docs only here; no hub change, no hub version bump. - audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis, including the run that produced a FALSE NEGATIVE and the mechanism behind it (the agent re-binds an unmounted drive within ~60s, so the drive gate's startup reconcile restarted the apps one second before the sweep looked). Records that the write hazard was blocked only by an ACCIDENTAL filesystem permission that no code owns and no test pins. - architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired state; the sweep observes a SETTLED fleet and each sample must refresh first; nothing is started without asking, fail-safe. Plus the durable warning: Manager.StartStack has no gate of its own. - 00-capability-map — the boot-recovery row, with the repeat count cited per N.5 (6 of 6 hard resets) rather than a bare PROVEN-LIVE. - OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW and closed the same session, marked a regression from v0.189.0. - STATUS.md — the power-cut line moved from "What's broken" to "What works right now" with its repeat count; one dated bullet in the change log. - CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is not a disproof; widening a window makes previously-unreachable overlaps reachable; and a settle detector is only as good as the freshness of what it samples — the fix's own defect, found live rather than by review.
This commit is contained in:
+50
@@ -164,6 +164,56 @@ Five decisions were deliberately **left open for the operator** and are recorded
|
||||
stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with
|
||||
the guest it backs up. Gaps minted the same session: **R-102 … R-108**.
|
||||
|
||||
**S-13 — boot recovery finished, and the lesson is about the DIAGNOSIS ORDER (controller v0.190.0,
|
||||
2026-08-02, R-157 A · R-170 · R-171).**
|
||||
|
||||
**The session's most valuable half hour was spent NOT writing code.** A hole was reasoned out of the
|
||||
v0.189.0 diff — replacing the container-count term with recorded intent should make a
|
||||
drive-gate-stopped app read as a boot orphan — and the task's own rule was to CONFIRM it on hardware
|
||||
before writing a fix. **The first attempt to confirm it produced a false negative**, and reporting
|
||||
that as a disproof would have been wrong: unmounting only the parent bind is healed by the agent
|
||||
within ~60 s, so the drive gate's startup reconcile restarted the apps **one second before** the
|
||||
sweep looked. `no boot-orphaned apps` in that log is a race that went the safe way, not a mechanism.
|
||||
Holding the drive genuinely absent reproduced it immediately. **"It didn't happen this time" is not a
|
||||
disproof — name the mechanism or run it again.**
|
||||
|
||||
**The confirmation also changed the severity, in both directions.** The write hazard did NOT
|
||||
materialise: compose failed `mkdir …/userdata: permission denied`, because the unbound mountpoint is
|
||||
host-root-owned and the guest is unprivileged. **That protection is accidental** — no code chose it,
|
||||
no test pins it, it is one `chown` or one privileged guest away from gone, and its removal would be
|
||||
invisible until data landed on the wrong disk. Meanwhile the harm that DID occur was real on every
|
||||
box and was not in the hypothesis: two wasted attempts and a **false dead-app alarm for an app the
|
||||
drive gate is deliberately holding**. Diagnosing first is what produced both facts.
|
||||
|
||||
**The fix was already in the codebase, on another path.** The API's `startGatedByMissingDrive`
|
||||
refuses a customer's start on an absent drive with a Hungarian message. The sweep bypassed it by
|
||||
calling `Manager.StartStack` directly. **`StartStack` has no gate of its own** — that is the durable
|
||||
fact worth carrying: every caller that is not the customer must decide for itself whether the app may
|
||||
run, and there are now fourteen of them.
|
||||
|
||||
**Widening a window makes previously-unreachable overlaps reachable, and that is a design input, not
|
||||
an afterthought.** The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a
|
||||
50 s window can. All three holders answer one seam rather than three, because they differ only in
|
||||
the reason string.
|
||||
|
||||
**A test rejected my first constant, and the comment now says so.** `settle + budget + one retry`
|
||||
must fit inside `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s
|
||||
against a 90 s grace. The budget is 50 s **because a test said so**, and the code records that rather
|
||||
than presenting the number as taste. Widening the grace to fit was rejected outright: it hides a late
|
||||
recovery instead of reporting one.
|
||||
|
||||
**AND THE FIX HAD ITS OWN DEFECT, FOUND BY LIVE VALIDATION AND NOT BY REVIEW.** The window sampled
|
||||
`GetStacks()` — the Manager's in-memory map, refreshed by the scheduler every 10 s — every 5 s. Two
|
||||
identical samples could therefore mean *the cache did not update*, not that the fleet had settled. It
|
||||
surfaced as a container removed ~5 s before the window closed still being in the sampled fleet, with
|
||||
the sweep logging `no boot-orphaned apps` for an app that had none. **Generalise it: a settle
|
||||
detector is only as good as the freshness of what it samples — if the source is cached, refresh it or
|
||||
you are watching the cache settle, not the system.**
|
||||
|
||||
**Live: 6/6 hard resets on the shipped build** (every app back; a customer-stopped app down in all
|
||||
six), window settle times 10/40/10/10/15/15 s — routinely 2–8× the old fixed 5 s. The sharpest
|
||||
evidence is a same-app before/after on one box: missed at 18:08:35, recovered at 18:18:50.
|
||||
|
||||
**S-12 — D-b is BUILT (controller v0.189.0, 2026-08-02, R-166).** The desired/in-flight/observed
|
||||
split now exists; the S-1 contract lives in `architecture/02-controller-module-map.md` §0a.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user