docs: R-157 A / R-170 / R-171 closed — boot recovery finished
gates / gates (push) Successful in 7s

Controller v0.190.0. Docs only here; no hub change, no hub version bump.

- audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis,
  including the run that produced a FALSE NEGATIVE and the mechanism behind it
  (the agent re-binds an unmounted drive within ~60s, so the drive gate's startup
  reconcile restarted the apps one second before the sweep looked). Records that
  the write hazard was blocked only by an ACCIDENTAL filesystem permission that no
  code owns and no test pins.
- architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired
  state; the sweep observes a SETTLED fleet and each sample must refresh first;
  nothing is started without asking, fail-safe. Plus the durable warning:
  Manager.StartStack has no gate of its own.
- 00-capability-map — the boot-recovery row, with the repeat count cited per N.5
  (6 of 6 hard resets) rather than a bare PROVEN-LIVE.
- OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW
  and closed the same session, marked a regression from v0.189.0.
- STATUS.md — the power-cut line moved from "What's broken" to "What works right
  now" with its repeat count; one dated bullet in the change log.
- CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is
  not a disproof; widening a window makes previously-unreachable overlaps
  reachable; and a settle detector is only as good as the freshness of what it
  samples — the fix's own defect, found live rather than by review.
This commit is contained in:
2026-08-02 20:38:21 +02:00
parent 5c97fbc397
commit ad28699761
7 changed files with 247 additions and 16 deletions
+50
View File
@@ -164,6 +164,56 @@ Five decisions were deliberately **left open for the operator** and are recorded
stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with
the guest it backs up. Gaps minted the same session: **R-102 … R-108**.
**S-13 — boot recovery finished, and the lesson is about the DIAGNOSIS ORDER (controller v0.190.0,
2026-08-02, R-157 A · R-170 · R-171).**
**The session's most valuable half hour was spent NOT writing code.** A hole was reasoned out of the
v0.189.0 diff — replacing the container-count term with recorded intent should make a
drive-gate-stopped app read as a boot orphan — and the task's own rule was to CONFIRM it on hardware
before writing a fix. **The first attempt to confirm it produced a false negative**, and reporting
that as a disproof would have been wrong: unmounting only the parent bind is healed by the agent
within ~60 s, so the drive gate's startup reconcile restarted the apps **one second before** the
sweep looked. `no boot-orphaned apps` in that log is a race that went the safe way, not a mechanism.
Holding the drive genuinely absent reproduced it immediately. **"It didn't happen this time" is not a
disproof — name the mechanism or run it again.**
**The confirmation also changed the severity, in both directions.** The write hazard did NOT
materialise: compose failed `mkdir …/userdata: permission denied`, because the unbound mountpoint is
host-root-owned and the guest is unprivileged. **That protection is accidental** — no code chose it,
no test pins it, it is one `chown` or one privileged guest away from gone, and its removal would be
invisible until data landed on the wrong disk. Meanwhile the harm that DID occur was real on every
box and was not in the hypothesis: two wasted attempts and a **false dead-app alarm for an app the
drive gate is deliberately holding**. Diagnosing first is what produced both facts.
**The fix was already in the codebase, on another path.** The API's `startGatedByMissingDrive`
refuses a customer's start on an absent drive with a Hungarian message. The sweep bypassed it by
calling `Manager.StartStack` directly. **`StartStack` has no gate of its own** — that is the durable
fact worth carrying: every caller that is not the customer must decide for itself whether the app may
run, and there are now fourteen of them.
**Widening a window makes previously-unreachable overlaps reachable, and that is a design input, not
an afterthought.** The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a
50 s window can. All three holders answer one seam rather than three, because they differ only in
the reason string.
**A test rejected my first constant, and the comment now says so.** `settle + budget + one retry`
must fit inside `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s
against a 90 s grace. The budget is 50 s **because a test said so**, and the code records that rather
than presenting the number as taste. Widening the grace to fit was rejected outright: it hides a late
recovery instead of reporting one.
**AND THE FIX HAD ITS OWN DEFECT, FOUND BY LIVE VALIDATION AND NOT BY REVIEW.** The window sampled
`GetStacks()` — the Manager's in-memory map, refreshed by the scheduler every 10 s — every 5 s. Two
identical samples could therefore mean *the cache did not update*, not that the fleet had settled. It
surfaced as a container removed ~5 s before the window closed still being in the sampled fleet, with
the sweep logging `no boot-orphaned apps` for an app that had none. **Generalise it: a settle
detector is only as good as the freshness of what it samples — if the source is cached, refresh it or
you are watching the cache settle, not the system.**
**Live: 6/6 hard resets on the shipped build** (every app back; a customer-stopped app down in all
six), window settle times 10/40/10/10/15/15 s — routinely 28× the old fixed 5 s. The sharpest
evidence is a same-app before/after on one box: missed at 18:08:35, recovered at 18:18:50.
**S-12 — D-b is BUILT (controller v0.189.0, 2026-08-02, R-166).** The desired/in-flight/observed
split now exists; the S-1 contract lives in `architecture/02-controller-module-map.md` §0a.