docs: R-157 A / R-170 / R-171 closed — boot recovery finished
gates / gates (push) Successful in 7s

Controller v0.190.0. Docs only here; no hub change, no hub version bump.

- audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis,
  including the run that produced a FALSE NEGATIVE and the mechanism behind it
  (the agent re-binds an unmounted drive within ~60s, so the drive gate's startup
  reconcile restarted the apps one second before the sweep looked). Records that
  the write hazard was blocked only by an ACCIDENTAL filesystem permission that no
  code owns and no test pins.
- architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired
  state; the sweep observes a SETTLED fleet and each sample must refresh first;
  nothing is started without asking, fail-safe. Plus the durable warning:
  Manager.StartStack has no gate of its own.
- 00-capability-map — the boot-recovery row, with the repeat count cited per N.5
  (6 of 6 hard resets) rather than a bare PROVEN-LIVE.
- OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW
  and closed the same session, marked a regression from v0.189.0.
- STATUS.md — the power-cut line moved from "What's broken" to "What works right
  now" with its repeat count; one dated bullet in the change log.
- CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is
  not a disproof; widening a window makes previously-unreachable overlaps
  reachable; and a settle detector is only as good as the freshness of what it
  samples — the fix's own defect, found live rather than by review.
This commit is contained in:
admin committed 2026-08-02 20:38:21 +02:00
1 parent 5c97fbc397
commit ad28699761
7 files changed
+247 -16

No files matched your search

@@ -80,7 +80,7 @@
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill **Route + RTO → `07-backup-architecture.md` §8 rows 6, 8, 9** — measured 84–112 s local / 1101 s PBS into a scratch guest; a restore to a DIFFERENT host is unmeasured |
| PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending |
| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→**v0.189.0**, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21; boot-orphan leg re-proven on the new signal 2026-08-02) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21** (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (`compose down` → 0 containers), calibre-web and bookstack left running, then `pct reboot 9201`: the gate recreated calibre-web and logged `1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`; immich came back **stopped**, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The **static-guest** half of the network leg stays deliberately out of scope → **R-50**. **2026-08-02 — the boot-orphan leg now rests on a RECORDED signal, not an inference (R-166, controller v0.189.0).** Both the R-52 sweep and the R-55 gate above decided "the customer stopped this" from *zero containers*, which is also what a power cut mid-compose and an interrupted deploy leave behind — so two real faults were read as deliberate stops and stranded silently (R-157 mechanism B). The customer's intent is now written to `app.yaml` (`desired_state`) by their own action and read directly. **PROVEN-LIVE on 9201** in three flows: a UI Stop persisted `stopped` and survived a controller restart with the app still down and NOT listed as a candidate; an app recorded `running` whose containers were removed out-of-band was recovered by name (`[bootrecon] 1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt) — **the case that was invisible before**; and a legacy `app.yaml` with no field was skipped exactly as before and was never inferred to be stopped. **Interrupted app-data operations are covered separately and are NOT proven-live** — `backup.AppStopGuard` restarts apps left stopped by a killed volume dump / offsite reconstitute / `.fab` export, and that leg is unit-proven + red-proofed only (killing the controller mid-backup on a live box was not exercised): **IMPLEMENTED, not PROVEN-LIVE.** **R-55's own gate was NOT migrated → R-170** |
| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→**v0.190.0**, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21; boot-orphan leg rebuilt and re-proven 2026-08-02 — **6 of 6 hard resets**, repeat count cited per N.5) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21** (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (`compose down` → 0 containers), calibre-web and bookstack left running, then `pct reboot 9201`: the gate recreated calibre-web and logged `1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`; immich came back **stopped**, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The **static-guest** half of the network leg stays deliberately out of scope → **R-50**. **2026-08-02 — the boot-orphan leg now rests on a RECORDED signal, not an inference (R-166, controller v0.189.0).** Both the R-52 sweep and the R-55 gate above decided "the customer stopped this" from *zero containers*, which is also what a power cut mid-compose and an interrupted deploy leave behind — so two real faults were read as deliberate stops and stranded silently (R-157 mechanism B). The customer's intent is now written to `app.yaml` (`desired_state`) by their own action and read directly. **PROVEN-LIVE on 9201** in three flows: a UI Stop persisted `stopped` and survived a controller restart with the app still down and NOT listed as a candidate; an app recorded `running` whose containers were removed out-of-band was recovered by name (`[bootrecon] 1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt) — **the case that was invisible before**; and a legacy `app.yaml` with no field was skipped exactly as before and was never inferred to be stopped. **Interrupted app-data operations are covered separately and are NOT proven-live** — `backup.AppStopGuard` restarts apps left stopped by a killed volume dump / offsite reconstitute / `.fab` export, and that leg is unit-proven + red-proofed only (killing the controller mid-backup on a live box was not exercised): **IMPLEMENTED, not PROVEN-LIVE.** **R-170 and R-171 closed the same day (controller v0.190.0).** **R-157 mechanism A** — the sweep observed ONCE at T+5 s, while docker was still restoring, and never re-checked (3 of 6 hard resets). It is now a settle-then-sweep window: sample every 5 s, settled after 3 identical samples, ONE sweep at the end, terminating on settled or a 50 s budget (sized so settle+budget+one retry stays inside the 90 s dead-app grace; a test rejected 60 s at 95 s). **Repeat count, per this map's own rule: 6 of 6 hard resets on the shipped build brought every app back, and an app the customer had stopped stayed down in all 6** (window settle times 10/40/10/10/15/15 s — i.e. it routinely waited 2–8× longer than the old fixed 5 s). A same-app before/after on one box is the sharpest evidence: the pre-fix window logged `no boot-orphaned apps` for calibre-web at 18:08:35; the fixed one found and recovered it at 18:18:50. **R-170** — `shouldRecreateOnBoot` now reads intent too, so the two boot gates agree; proven live in one reboot (calibre-web `running`+zero containers recreated, immicLine truncated
| Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) |
| An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 |
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
@@ -120,10 +120,44 @@ Applied: a lost or corrupt marker means the app is not auto-restarted **by that
the pre-v0.189.0 position, not a new hazard. A lost `app.yaml` already means the app is not deployed.
**Nothing here may make an absent file more dangerous than a present one.**
### Not migrated
### Boot recovery reads desired, and asks before it acts (v0.190.0)
`shouldRecreateOnBoot` (`internal/web/intermediary.go`) — the drive-backed boot gate — still derives
desired from observed via `hasContainers`. Tracked as **R-170**.
**Both boot gates now read desired state.** `bootrecon.isBootOrphan` (the R-52 sweep) and
`shouldRecreateOnBoot` (the drive-backed recreate gate) answer the same question — *did the customer
want this running?* — with the same three-way table, absent falling back to the pre-v0.190.0
container count in both. Their agreement is pinned from both sides against one fixture table, because
an import cycle prevents testing them together. R-170 closed the last gate that still guessed.
**The sweep observes a SETTLED fleet, not a single early sample.** It samples (name, state, container
count) every 5 s, calls the fleet settled after 3 identical samples, and sweeps **once**, at the end.
The window ends on settled or a 50 s budget, and the log says which. Two constraints bound it:
- `bootReconcileSettle + budget + one DefaultRetryDelay` must stay inside `deadAppBootGrace`, or a
successful recovery stops being silent. This is arithmetic, pinned by a test.
- **each sample must REFRESH first.** `GetStacks()` is the Manager's in-memory map, refreshed by the
scheduler on its own 10 s cadence; sampling it faster without refreshing lets "settled" mean "the
cache did not update". Found by live validation, not review.
A recovery completing after the grace emits a `LATE RECOVERY` warning naming the apps. The grace is
**never** widened to make a late recovery look silent.
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
answer it, and the last two only became reachable once the window widened past T+5 s:
| Holder | Why starting would be wrong |
|---|---|
| the drive-absent gate | compose creates bind sources wherever the mountpoint points — the guest rootfs |
| a quiesce | a running app inside a snapshot meant to be clean-shutdown-consistent |
| an in-flight app-data operation | restarting an app under its own tar |
The rule is **not new** — the API's `startGatedByMissingDrive` already refused a customer's start on
an absent drive. The sweep bypassed it by calling `Manager.StartStack` directly, which is what R-171
closed. Held apps are reported separately from "still down": they are not a fault the sweep failed to
fix, and reporting them as one is a false alarm.
**Read this before adding a fourth caller of `Manager.StartStack`.** That method has no gate of its
own; every caller that is not the customer must decide for itself whether the app may run.
---