diff --git a/CONTEXT.md b/CONTEXT.md index b2a035f..5a27202 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -164,6 +164,56 @@ Five decisions were deliberately **left open for the operator** and are recorded stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with the guest it backs up. Gaps minted the same session: **R-102 … R-108**. +**S-13 — boot recovery finished, and the lesson is about the DIAGNOSIS ORDER (controller v0.190.0, +2026-08-02, R-157 A · R-170 · R-171).** + +**The session's most valuable half hour was spent NOT writing code.** A hole was reasoned out of the +v0.189.0 diff — replacing the container-count term with recorded intent should make a +drive-gate-stopped app read as a boot orphan — and the task's own rule was to CONFIRM it on hardware +before writing a fix. **The first attempt to confirm it produced a false negative**, and reporting +that as a disproof would have been wrong: unmounting only the parent bind is healed by the agent +within ~60 s, so the drive gate's startup reconcile restarted the apps **one second before** the +sweep looked. `no boot-orphaned apps` in that log is a race that went the safe way, not a mechanism. +Holding the drive genuinely absent reproduced it immediately. **"It didn't happen this time" is not a +disproof — name the mechanism or run it again.** + +**The confirmation also changed the severity, in both directions.** The write hazard did NOT +materialise: compose failed `mkdir …/userdata: permission denied`, because the unbound mountpoint is +host-root-owned and the guest is unprivileged. **That protection is accidental** — no code chose it, +no test pins it, it is one `chown` or one privileged guest away from gone, and its removal would be +invisible until data landed on the wrong disk. Meanwhile the harm that DID occur was real on every +box and was not in the hypothesis: two wasted attempts and a **false dead-app alarm for an app the +drive gate is deliberately holding**. Diagnosing first is what produced both facts. + +**The fix was already in the codebase, on another path.** The API's `startGatedByMissingDrive` +refuses a customer's start on an absent drive with a Hungarian message. The sweep bypassed it by +calling `Manager.StartStack` directly. **`StartStack` has no gate of its own** — that is the durable +fact worth carrying: every caller that is not the customer must decide for itself whether the app may +run, and there are now fourteen of them. + +**Widening a window makes previously-unreachable overlaps reachable, and that is a design input, not +an afterthought.** The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a +50 s window can. All three holders answer one seam rather than three, because they differ only in +the reason string. + +**A test rejected my first constant, and the comment now says so.** `settle + budget + one retry` +must fit inside `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s +against a 90 s grace. The budget is 50 s **because a test said so**, and the code records that rather +than presenting the number as taste. Widening the grace to fit was rejected outright: it hides a late +recovery instead of reporting one. + +**AND THE FIX HAD ITS OWN DEFECT, FOUND BY LIVE VALIDATION AND NOT BY REVIEW.** The window sampled +`GetStacks()` — the Manager's in-memory map, refreshed by the scheduler every 10 s — every 5 s. Two +identical samples could therefore mean *the cache did not update*, not that the fleet had settled. It +surfaced as a container removed ~5 s before the window closed still being in the sampled fleet, with +the sweep logging `no boot-orphaned apps` for an app that had none. **Generalise it: a settle +detector is only as good as the freshness of what it samples — if the source is cached, refresh it or +you are watching the cache settle, not the system.** + +**Live: 6/6 hard resets on the shipped build** (every app back; a customer-stopped app down in all +six), window settle times 10/40/10/10/15/15 s — routinely 2–8× the old fixed 5 s. The sharpest +evidence is a same-app before/after on one box: missed at 18:08:35, recovered at 18:18:50. + **S-12 — D-b is BUILT (controller v0.189.0, 2026-08-02, R-166).** The desired/in-flight/observed split now exists; the S-1 contract lives in `architecture/02-controller-module-map.md` §0a. diff --git a/STATUS.md b/STATUS.md index 5039d3b..75fa30c 100644 --- a/STATUS.md +++ b/STATUS.md @@ -16,15 +16,12 @@ home network, and open apps from a launcher or a shared link. Backups run on the places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can restore files and app data from the drive alone. Proven end to end on real hardware. -## What's broken +**Apps come back after a power cut.** The machine tells an app the customer switched off from one +that simply did not come back, and waits for the system to finish starting before deciding instead of +glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every +time, and an app switched off deliberately stayed off every time. -**After a power cut, an app can stay switched off — and the machine may look too early to notice.** -It now knows the difference between an app the customer switched off and one that simply didn't come -back: the customer's choice is written down, so an app meant to be running is started again however -it went missing. What is left is timing — it looks about five seconds after starting, while the -system is still coming up, and never looks again. *(R-157; the "decided the customer switched it off" -half is fixed — R-166, 2026-08-02. The same guess still lives in one other place, for apps kept on a -plug-in drive — R-170.)* +## What's broken **The off-site copy can be erased by the machine that made it** — the credential that writes it can also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy. @@ -69,6 +66,15 @@ It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)* ## Changed since last update +- **2026-08-02** — Boot recovery finished. Both halves of the power-cut problem are closed: the + machine records what the customer asked for, and it now waits for the system to finish starting + before deciding what is missing. Six hard resets in a row, everything back every time. A hole the + previous day's change had opened — the machine would have tried to start an app whose external + drive was missing — was found by reading the code, reproduced on the demo box **before** anything + was changed, and fixed the same day. No data ever reached the wrong disk, but only because of a + filesystem accident nothing was relying on deliberately, which is why it was fixed rather than + noted. + - **2026-08-02** — Fixed and then doubled: thirteen mechanical checks had built up across the four repositories and nothing ran most of them; two were failing quietly, one since 14 July. Both fixed. Every repository now has one command that runs all its checks, it runs automatically before every diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 548f42f..f7ceec3 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -80,7 +80,7 @@ | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D | | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill **Route + RTO → `07-backup-architecture.md` §8 rows 6, 8, 9** — measured 84–112 s local / 1101 s PBS into a scratch guest; a restore to a DIFFERENT host is unmeasured | | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending | -| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→**v0.189.0**, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21; boot-orphan leg re-proven on the new signal 2026-08-02) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21** (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (`compose down` → 0 containers), calibre-web and bookstack left running, then `pct reboot 9201`: the gate recreated calibre-web and logged `1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`; immich came back **stopped**, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The **static-guest** half of the network leg stays deliberately out of scope → **R-50**. **2026-08-02 — the boot-orphan leg now rests on a RECORDED signal, not an inference (R-166, controller v0.189.0).** Both the R-52 sweep and the R-55 gate above decided "the customer stopped this" from *zero containers*, which is also what a power cut mid-compose and an interrupted deploy leave behind — so two real faults were read as deliberate stops and stranded silently (R-157 mechanism B). The customer's intent is now written to `app.yaml` (`desired_state`) by their own action and read directly. **PROVEN-LIVE on 9201** in three flows: a UI Stop persisted `stopped` and survived a controller restart with the app still down and NOT listed as a candidate; an app recorded `running` whose containers were removed out-of-band was recovered by name (`[bootrecon] 1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt) — **the case that was invisible before**; and a legacy `app.yaml` with no field was skipped exactly as before and was never inferred to be stopped. **Interrupted app-data operations are covered separately and are NOT proven-live** — `backup.AppStopGuard` restarts apps left stopped by a killed volume dump / offsite reconstitute / `.fab` export, and that leg is unit-proven + red-proofed only (killing the controller mid-backup on a live box was not exercised): **IMPLEMENTED, not PROVEN-LIVE.** **R-55's own gate was NOT migrated → R-170** | +| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→**v0.190.0**, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21; boot-orphan leg rebuilt and re-proven 2026-08-02 — **6 of 6 hard resets**, repeat count cited per N.5) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21** (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (`compose down` → 0 containers), calibre-web and bookstack left running, then `pct reboot 9201`: the gate recreated calibre-web and logged `1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`; immich came back **stopped**, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The **static-guest** half of the network leg stays deliberately out of scope → **R-50**. **2026-08-02 — the boot-orphan leg now rests on a RECORDED signal, not an inference (R-166, controller v0.189.0).** Both the R-52 sweep and the R-55 gate above decided "the customer stopped this" from *zero containers*, which is also what a power cut mid-compose and an interrupted deploy leave behind — so two real faults were read as deliberate stops and stranded silently (R-157 mechanism B). The customer's intent is now written to `app.yaml` (`desired_state`) by their own action and read directly. **PROVEN-LIVE on 9201** in three flows: a UI Stop persisted `stopped` and survived a controller restart with the app still down and NOT listed as a candidate; an app recorded `running` whose containers were removed out-of-band was recovered by name (`[bootrecon] 1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt) — **the case that was invisible before**; and a legacy `app.yaml` with no field was skipped exactly as before and was never inferred to be stopped. **Interrupted app-data operations are covered separately and are NOT proven-live** — `backup.AppStopGuard` restarts apps left stopped by a killed volume dump / offsite reconstitute / `.fab` export, and that leg is unit-proven + red-proofed only (killing the controller mid-backup on a live box was not exercised): **IMPLEMENTED, not PROVEN-LIVE.** **R-170 and R-171 closed the same day (controller v0.190.0).** **R-157 mechanism A** — the sweep observed ONCE at T+5 s, while docker was still restoring, and never re-checked (3 of 6 hard resets). It is now a settle-then-sweep window: sample every 5 s, settled after 3 identical samples, ONE sweep at the end, terminating on settled or a 50 s budget (sized so settle+budget+one retry stays inside the 90 s dead-app grace; a test rejected 60 s at 95 s). **Repeat count, per this map's own rule: 6 of 6 hard resets on the shipped build brought every app back, and an app the customer had stopped stayed down in all 6** (window settle times 10/40/10/10/15/15 s — i.e. it routinely waited 2–8× longer than the old fixed 5 s). A same-app before/after on one box is the sharpest evidence: the pre-fix window logged `no boot-orphaned apps` for calibre-web at 18:08:35; the fixed one found and recovered it at 18:18:50. **R-170** — `shouldRecreateOnBoot` now reads intent too, so the two boot gates agree; proven live in one reboot (calibre-web `running`+zero containers recreated, immich `stopped` left alone). **R-171 — a regression v0.189.0 introduced, found by reading the diff and CONFIRMED on hardware before any fix was written:** the sweep started an app whose drive was absent, burned both attempts and raised a false dead-app alarm. The *write* hazard was blocked only by an ACCIDENTAL filesystem permission (host-root-owned mountpoint + unprivileged guest) that no code owns and no test pinned — which is why it was fixed rather than noted. New fail-safe `bootrecon.StartGate` (cannot determine ⇒ do not start), also covering quiesce and in-flight app-data operations. **One defect in the fix itself, found by live validation and not by review:** the window sampled the Manager's 10 s-refreshed cache, so "settled" could mean "the cache did not update"; `sampleBootFleet` now refreshes first. Evidence: `audits/DIAG-bootrecon-drive-absent-2026-08-02.md`, `felhom-controller/REPORT.md` | | Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) | | An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 | | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped | diff --git a/documentation/architecture/02-controller-module-map.md b/documentation/architecture/02-controller-module-map.md index f1bd2f9..4231286 100644 --- a/documentation/architecture/02-controller-module-map.md +++ b/documentation/architecture/02-controller-module-map.md @@ -120,10 +120,44 @@ Applied: a lost or corrupt marker means the app is not auto-restarted **by that the pre-v0.189.0 position, not a new hazard. A lost `app.yaml` already means the app is not deployed. **Nothing here may make an absent file more dangerous than a present one.** -### Not migrated +### Boot recovery reads desired, and asks before it acts (v0.190.0) -`shouldRecreateOnBoot` (`internal/web/intermediary.go`) — the drive-backed boot gate — still derives -desired from observed via `hasContainers`. Tracked as **R-170**. +**Both boot gates now read desired state.** `bootrecon.isBootOrphan` (the R-52 sweep) and +`shouldRecreateOnBoot` (the drive-backed recreate gate) answer the same question — *did the customer +want this running?* — with the same three-way table, absent falling back to the pre-v0.190.0 +container count in both. Their agreement is pinned from both sides against one fixture table, because +an import cycle prevents testing them together. R-170 closed the last gate that still guessed. + +**The sweep observes a SETTLED fleet, not a single early sample.** It samples (name, state, container +count) every 5 s, calls the fleet settled after 3 identical samples, and sweeps **once**, at the end. +The window ends on settled or a 50 s budget, and the log says which. Two constraints bound it: + +- `bootReconcileSettle + budget + one DefaultRetryDelay` must stay inside `deadAppBootGrace`, or a + successful recovery stops being silent. This is arithmetic, pinned by a test. +- **each sample must REFRESH first.** `GetStacks()` is the Manager's in-memory map, refreshed by the + scheduler on its own 10 s cadence; sampling it faster without refreshing lets "settled" mean "the + cache did not update". Found by live validation, not review. + +A recovery completing after the grace emits a `LATE RECOVERY` warning naming the apps. The grace is +**never** widened to make a late recovery look silent. + +**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question +the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders +answer it, and the last two only became reachable once the window widened past T+5 s: + +| Holder | Why starting would be wrong | +|---|---| +| the drive-absent gate | compose creates bind sources wherever the mountpoint points — the guest rootfs | +| a quiesce | a running app inside a snapshot meant to be clean-shutdown-consistent | +| an in-flight app-data operation | restarting an app under its own tar | + +The rule is **not new** — the API's `startGatedByMissingDrive` already refused a customer's start on +an absent drive. The sweep bypassed it by calling `Manager.StartStack` directly, which is what R-171 +closed. Held apps are reported separately from "still down": they are not a fault the sweep failed to +fix, and reporting them as one is a false alarm. + +**Read this before adding a fourth caller of `Manager.StartStack`.** That method has no gate of its +own; every caller that is not the customer must decide for itself whether the app may run. --- diff --git a/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md b/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md new file mode 100644 index 0000000..66fe7f0 --- /dev/null +++ b/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md @@ -0,0 +1,139 @@ +# DIAG — does the R-166 boot sweep start an app whose data drive is absent? + +**Date:** 2026-08-02 · **Box:** demo `felhom-pve` guest 9201 · **Controller:** v0.189.0 +**Question raised by:** reading the v0.189.0 diff, **not** by an incident. Diagnosed before any fix +was written (task §9.2, "diagnose before theorising"). + +## Verdict + +**CONFIRMED — the sweep starts it.** Observed directly, twice in the same run (both attempts). + +**With a qualification that changes the severity but not the fix:** on this box the resulting +`docker compose up -d` **failed** and **nothing was written** to the wrong disk. The mechanism that +prevented the write is a **filesystem-permission accident that no code owns**, and the harm that DID +occur is a different one: a false dead-app alarm for an app the drive gate is deliberately holding. + +## The reasoning under test + +`bootrecon` imports `stacks` alone and has no storage awareness. Since v0.189.0 `isBootOrphan` +returns true for `desired_state: running` + zero containers. The drive-absent gate stops apps with +`compose down` (leaving exactly zero containers) and never touches `desired_state`, because it is not +the customer. `Manager.StartStack` has no drive gate. Therefore the sweep should start an app whose +drive is absent. + +## Method + +The drive is a real USB disk (`/dev/sdb`), bound under the stable parent at +`/mnt/felhom-drives/hdd_1`. The gate's absence signal is the agent's `BoundUnderParent` +(`planDriveGates`, `intermediary.go:226`), so the drive was made absent **by unmounting it**, not by +editing controller state — the state edit would have proven a different thing. + +**Run 1 was contaminated and is reported because it produced a mechanism.** Unmounting only the +parent bind was not enough: **the agent re-binds it within ~60 s** while `/dev/sdb` is still mounted +at `/mnt/hdd_1`. In that run the drive gate's startup reconcile re-attached the drive and restarted +the apps at `17:18:47`, **one second before** `bootrecon` looked at `17:18:48` — which then logged +`no boot-orphaned apps`. That is a race that happened to go the safe way, **not** a disproof. Had it +been reported as one, the conclusion would have been wrong. + +Run 2 therefore unmounted **both** `/mnt/felhom-drives/hdd_1` and `/mnt/hdd_1`, and held them +unmounted against the agent's healing for the duration. + +## Preconditions, all verified before the observation + +| # | Precondition | Observed | +|---|---|---| +| 1 | drive absent | `mount \| grep -c hdd_1` → **0** | +| 2 | app has zero containers | `docker ps -a --filter name=calibre` → **0** | +| 3 | intent still says running | `desired_state: running` | +| 4 | app still deployed | `deployed: true` | + +The gate stopped the apps itself first, on its own signal: + +``` +17:22:07 [WARN] [gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 2 app(s): [calibre-web immich] +``` + +`immich` was set to `desired_state: stopped` beforehand to scope the blast radius to one app; the +gate is per-drive, so it stops every app on that drive regardless. + +## The observation + +``` +17:22:29 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded +17:22:34 [INFO] [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s) +17:22:34 [INFO] [stacks] Starting stack: calibre-web +17:22:35 [WARN] [bootrecon] attempt 1/2: start "calibre-web" failed after 0.4s: exit code 1 +17:23:05 [WARN] [bootrecon] attempt 2/2: start "calibre-web" failed after 0.4s: exit code 1 +17:23:05 [WARN] [bootrecon] gave up after 2 attempt(s): recovered=[] still down=[calibre-web] (the dead-app alarm now owns these) +``` + +**The sweep selected a drive-absent app and called `StartStack` on it.** That is the confirmation. + +## What stopped the write, and why it must not be relied on + +``` +Error response from daemon: error while creating mount source path + '/mnt/felhom-drives/hdd_1/userdata/media/books': + mkdir /mnt/felhom-drives/hdd_1/userdata: permission denied +``` + +With the drive unbound, `/mnt/felhom-drives/hdd_1` is an empty directory on the host's `pve-root`, +and it is **host-root-owned**: + +``` +/mnt/felhom-drives uid=0 gid=0 mode=755 +/mnt/felhom-drives/hdd_1 uid=0 gid=0 mode=755 +``` + +Guest 9201 is an **unprivileged** LXC, so its container root is uid 100000 and cannot `mkdir` there. +Confirmed no write occurred: `find /mnt/felhom-drives/hdd_1/` returned the directory alone, and +`df -h /` was unchanged at `25G used / 28%` before and after. + +**This protection is accidental.** Nothing in the controller chose it, no test pins it, and it rests +on two conditions that are not guaranteed and are not checked anywhere: + +1. the guest is unprivileged (a privileged guest maps root→0 and the `mkdir` succeeds); +2. the stable-parent mountpoint directory is root-owned. When the drive **is** bound, that same path + shows `uid=100000 gid=100000` — i.e. guest-writable. Any code path or agent version that + pre-creates the mountpoint with guest ownership removes the protection silently. + +It is one `chown` away from being gone, and its removal would be invisible until data landed on the +wrong disk. It is therefore **not** a reason to leave the sweep unguarded. + +## The harm that DID occur + +Independent of the write question, and real on every box: + +- the sweep burns **both** attempts and 30 s of retry delay on an app that cannot start by design; +- it then hands the app to the **dead-app alarm** — `still down=[calibre-web] (the dead-app alarm now + owns these)` — producing a **false alarm about an app the drive gate is deliberately holding**, + which is exactly the noise class R-97/F-A1 exist to prevent; +- it leaves a `Created` container behind on each attempt. + +Before v0.189.0 none of this happened: `isBootOrphan` required `len(Containers) > 0`, and a +gate-stopped app has zero. **This is a regression introduced by v0.189.0.** + +## A second asymmetry found while restoring the box + +The **API** start path already refuses this correctly. Restoring `immich` through the endpoint the UI +calls, while the path was still flagged disconnected, returned: + +``` +{"ok":false,"error":"A(z) /mnt/felhom-drives/hdd_1 tárhely jelenleg nem elérhető — + az alkalmazás nem indítható, amíg a meghajtó vissza nem csatlakozik."} +``` + +That is `startGatedByMissingDrive` (`internal/api/router.go`). So the controller already holds the +rule "do not start an app whose drive is missing" — it is enforced on the customer's path and +**bypassed by the sweep**, which calls `Manager.StartStack` directly. The fix is to give the sweep +the same question to ask, not to invent a new rule. + +## Consequence for the task + +**Part 3 applies and is implemented first**, before the sweep's window is widened — a wider window +makes both the false alarm and the (currently accident-blocked) write hazard wider. Fail-safe +direction per §8.4: **cannot determine drive liveness → do not start.** Not starting is recoverable +(the gate's `Return` branch restarts the app when the drive comes back, and the alarm reports it +meanwhile); starting on an absent drive is not recoverable by anything automatic. + +Register row: **R-171**, marked as a regression from v0.189.0. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 17e80ca..7db0630 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -85,8 +85,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-138** | **A shared-zone `cf_api_token` is a zone-wide DNS-write capability on a customer's box** — written 0600 to `/opt/docker/stacks/traefik/.env` (`controller/internal/infra/infra.go:123`) | READY (S) | — | Today each box holds a token for a zone nobody else uses, so the blast radius is one customer. Under a shared customer zone, one compromised tester box could repoint every other tester's DNS. The ACME path is already switchable — an empty token selects HTTP-01 (`traefik.yml.tmpl`) — so the fix is policy plus a guard that refuses to hand a shared-zone customer a zone-scoped token. Same audit §5.2 | CC | | **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC | | **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. **THE REFERRAL IS RESOLVED, 2026-08-02 — papra is deployed NOWHERE, so the template fix strands nothing and can be applied.** The referral existed because changing where the volume mounts moves live data: an installed papra writes `db.sqlite` into the container's writable layer, and a remount relocates the path out from under it. With no instance deployed there is no live data to move, so the cheaper leg — **the template mounts `/app/app-data`** — is takeable directly, without waiting on upstream to adopt `/app/data`. **Provenance, stated because it decides the row:** the observation is `docker ps -a` on demo-hp's **guest 9201** returning empty, supplied with the 2026-08-02 task; **this session did not re-measure** (documentation-only, every box fenced). **Scope of that evidence, honestly:** it covers guest 9201 — the guest papra was convicted on in Campaign 10 — and **no other customer's guest was enumerated**, so a re-check belongs in the task that edits the template, before it edits it. **Next action: apply the template fix in `app-catalog-felhom.eu` and re-run `scripts/catalog_gates.py`** (deliberately not done here — that repo was out of scope for this task). See R-161 (nothing runs the gate automatically) and R-159/R-160 | CC | -| **R-157** | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms. MECHANISM B IS CLOSED; MECHANISM A REMAINS.** **(A) STILL OPEN:** the sweep runs ~5 s after controller start (`cmd/controller/main.go`, `runBootReconcile`) while docker is still restoring, sees "no boot-orphaned apps", and never re-checks — **intermittent, 3 of 6 hard resets**. R-166 did **not** touch the timing, so this is untouched and is now the whole of R-157. **(B) CLOSED by R-166 (controller v0.189.0, 2026-08-02):** an interruption mid-compose left the stack with **zero containers**, the signature bootrecon deliberately skipped as a user's Stop. The sweep now reads the customer's RECORDED intent (`desired_state` in `app.yaml`), so `running` + zero containers is an orphan and is recovered — **proven live on 9201**: containers removed out-of-band, controller restarted, `[bootrecon] 1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt. | **READY (S) — A only** | — | **A is now a pure timing fix and is the only thing left here:** a settle condition before the single observation, or one re-evaluation at the end of `deadAppBootGrace`, still bounded to the same attempts. Test the **consequence** (app running again before the grace expires), not the mechanism. Note that A and B were independent: B was a wrong signal, A is a premature observation, and B's fix makes A **more** visible (the sweep now has more it could recover, so observing too early costs more). Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`; B's closure in `felhom-controller/REPORT.md` (2026-08-02) | CC | -| **R-170** | **The drive-backed boot gate still infers a customer's Stop from a container count — R-157 mechanism B, one gate over, for drive-backed apps only.** `shouldRecreateOnBoot` (`controller/internal/web/intermediary.go:131`) ends in `&& hasContainers`, sampled at `:425` as `len(st.Containers) > 0`. That term was added deliberately by **R-55** (controller v0.157.0) to stop the gate resurrecting an app the customer had stopped, and it was the right fix with the signal available then. R-166 replaced exactly that inference in `bootrecon` with the recorded `desired_state`, and **this gate was left on the old signal** — so for a drive-backed app, a power cut or interrupted deploy that leaves zero containers is still read as a deliberate stop by this path | **READY (S) — NEW 2026-08-02** | — | **NOT acted on by R-166, deliberately** — the task scoped `bootrecon` and named this gate nowhere, and changing a second boot path in the same release would have shipped two behaviour changes under one live validation. **Consequence is bounded, not zero:** `bootrecon` now recovers the same app a few seconds later on intent, so the observable defect is narrowed to the gate's own recreate-onto-the-drive step being skipped — which matters when the app needs recreating against a re-bound drive, not merely starting. **Fix shape:** replace `hasContainers` with `stacks.DesiredStateOf(st) != DesiredStateStopped` and keep the legacy fallback, i.e. the §8.1 table this gate already half-implements. **Established free:** `grep -ro "R-170\b" documentation/ *.md` → 0 hits (2026-08-02) | CC | +| **R-157** | ~~**`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (B: controller v0.189.0; A: v0.190.0, 2026-08-02) | — | **Both mechanisms closed. (B)** the container-count signal → recorded intent (R-166). **(A)** the sweep looked ONCE at T+5 s, deriving candidates from a fleet docker was still restoring — 3 of 6 hard resets. Now a **settle-then-sweep window**: sample the fleet every 5 s, settled after 3 identical samples, sweep ONCE at the end; ends on settled OR a 50 s budget, and the log says which. **The budget is 50 s because a test rejected 60 s**: settle+budget+one 30 s retry must stay under the 90 s `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s. A window that genuinely overruns emits a `LATE RECOVERY` WARN naming the apps — the grace was NOT widened to hide it (§8.3). **A defect in the fix, found by live validation not review:** `GetStacks()` is the Manager's cache, refreshed by the scheduler every 10 s, so sampling it every 5 s without refreshing let "settled" mean "the cache did not update" — observed missing a container removed 5 s before the window closed. `sampleBootFleet` now refreshes first. **Live: 6/6 hard resets on the shipped build, every app back every time** (settle times 10/40/10/10/15/15 s — i.e. the window routinely waited 2–8× longer than the old fixed 5 s), plus a before/after on ONE app on ONE box: the pre-fix window logged `no boot-orphaned apps` for calibre-web at 18:08:35, the fixed one found and recovered it at 18:18:50 | — | +| **R-170** | ~~**The drive-backed boot gate infers a customer's Stop from a container count.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | `shouldRecreateOnBoot` now reads `desired_state` with the SAME three-way table as `isBootOrphan`: `stopped` → never; `running` → recreate whatever the container count; **absent → exactly the pre-v0.190.0 `hasContainers` behaviour**. `presentStable` untouched and still load-bearing (an absent drive is never recreated here — the very term the boot sweep was missing, R-171). Its comment argued at length FOR the container count and was rewritten; a correct implementation under a comment arguing the opposite is worse than either alone. **The agreement is pinned from BOTH sides** against one fixture table (`TestBothBootGatesAgreeOnIntent` / `TestShouldRecreateOnBoot_AgreesWithBootrecon`) because the two gates cannot be called from one package without an import cycle. **Live on 9201, both halves in one reboot:** calibre-web (drive-backed, `running`, ZERO containers) → `recreating drive-backed app calibre-web`; immich (`stopped`) → `1 drive-backed app(s) left stopped on purpose` | — | +| **R-171** | **The boot sweep started apps whose data drive was ABSENT — a regression introduced by v0.189.0, now FIXED.** Replacing `isBootOrphan`'s container-count term with recorded intent made a drive-gate-stopped app (`compose down` ⇒ zero containers, and the gate never touches `desired_state` because it is not the customer) read as a boot orphan | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | **Reasoned from the diff, then CONFIRMED on hardware before any fix was written** (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The sweep found and started calibre-web with its drive unmounted, burned both attempts and handed it to the dead-app alarm — **a false alarm about an app the drive gate is deliberately holding**. The *write* hazard did NOT materialise: compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is host-root-owned and the guest is unprivileged — **an accidental protection no code owns, no test pins, and one `chown` or one privileged guest away from gone**. Fix: new consumer-side seam `bootrecon.StartGate`, **fail-safe (cannot determine ⇒ do not start)**, wired in `main.go`; `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two cannot drift. **The rule is not new** — the API's `startGatedByMissingDrive` already refused this to the customer; the sweep bypassed it by calling `Manager.StartStack` directly. Widening the window (R-157 A) made two more holders reachable, so the same seam also refuses an app held by a **quiesce** or an **in-flight app-data operation** (§8.2), reusing `quiesce.SuppressedStacks()` and a new read-only `AppStopGuard.HeldStacks()`. Held apps report as `HeldByDrive`, never `StillDown` — that is the alarm's bucket. **ID established free:** `grep -ro "R-171\b" documentation/ *.md` → 0 hits before minting | — | | **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC | | **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `_` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC | | **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 6786ec6..65340be 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -162,9 +162,10 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the | R-154 | **`[first-boot]` is automated-install-only, and nothing in the tree said so** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | A PVE property, measured with a same-image control (`audits/SPIKE-universal-iso-3-2026-07-31.md` §2); recorded at `scripts/iso/pkg/build-deb.sh:6-11`. Superseded in practice by the `.deb` delivery route | | R-155 | **`iso-repack.sh` refused any ISO without `auto-installer-mode.toml`** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | **Narrowed, not deleted** — unchanged for `FELHOM_MENU=single` (`iso-repack.sh:121-128`), does not apply to `release` where the file's absence *is* gate G1. Do not remove it wholesale | | R-156 | **An app's data is neither persisted nor backed up, and it reports healthy** — a template mounts a volume at a path the app never writes, so data sits in the container's writable layer: lost on redeploy, tarred nightly as an empty dir, healthcheck green | S | **DETECTED + GATE SHIPPED; papra REFERRED** (catalog sweep, 2026-08-02) | Found by Campaign 10 on **papra**; the 53-template sweep convicted **gramps-web** and **wishlist** too and **fixed both**. The gate is `app-catalog-felhom.eu/scripts/check-volume-persistence.py` — a runtime probe (`docker diff` + mount occupancy + writability, canary self-test, fails closed); the detector and the gate are one program. **papra is NOT fixed**: the fix needs the app to use `/app/data` or the template to mount `/app/app-data`, so it is referred. Sweep evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` (53 probe.json). Ranking in `OPEN-ITEMS.md` | -| R-157 | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — two mechanisms** | S | **B SHIPPED (controller v0.189.0, 2026-08-02) — A still READY** | R-52 built `internal/bootrecon` for exactly this (F5: two apps `Exited` ~18 h). **(B) CLOSED by R-166:** a mid-compose interruption left **zero containers**, the signature bootrecon skipped as a user Stop; the sweep now reads the recorded `desired_state` instead of counting containers, so `running` + zero containers is an orphan — **proven live on 9201** (containers removed out-of-band → `1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt). **(A) UNTOUCHED and now the whole item:** the sweep fires ~5 s after start while docker is still restoring and never re-checks — **3 of 6 hard resets**, intermittent, which is what a race predicts and a wrong predicate does not. R-166 deliberately did not change the timing. **B's fix makes A cost more**, since the sweep now has more it could legitimately recover. Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` | +| R-157 | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover** | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.189.0 + v0.190.0, 2026-08-02)** | Both mechanisms closed. **B:** the container count → recorded intent (R-166). **A:** the single T+5 s observation → a settle-then-sweep window (sample every 5 s, settled after 3 identical samples, one sweep at the end, ends on settled OR a 50 s budget). Budget is 50 s because settle+budget+one retry must fit the 90 s `deadAppBootGrace`; a test rejected 60 s at 95 s. Late recoveries are REPORTED, not hidden by widening the grace. **The fix had its own defect, found live:** sampling the Manager's 10 s-refreshed cache let "settled" mean "the cache did not update" — `sampleBootFleet` now refreshes first. **6/6 hard resets clean on the shipped build**, plus a same-app before/after (missed 18:08:35, recovered 18:18:50) | +| R-170 | **The drive-backed boot gate infers a Stop from a container count** | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | `shouldRecreateOnBoot` reads `desired_state` with the same three-way table as `isBootOrphan`; absent keeps the old `hasContainers` behaviour exactly; `presentStable` untouched. Agreement pinned from both sides against one fixture table. Live: calibre-web (`running`, zero containers) recreated and immich (`stopped`) left alone in the same reboot | +| R-171 | **The boot sweep started apps whose data drive was ABSENT** — a regression from v0.189.0 | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | Reasoned from the diff, CONFIRMED on hardware first (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The write hazard was blocked only by an ACCIDENTAL filesystem permission (host-root-owned mountpoint + unprivileged guest); the false dead-app alarm was real on every box. New fail-safe `bootrecon.StartGate` seam, also covering quiesce and in-flight app-data operations (§8.2). The rule already existed on the API path (`startGatedByMissingDrive`); the sweep bypassed it | | R-166 | **App state gets a desired/observed model with its own store** (operator decision D-b) | M | **SHIPPED + PROVEN-LIVE — controller v0.189.0, 2026-08-02** | Tri-state `desired_state` in `app.yaml`, written ONLY by the customer's action; `bootrecon` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN** so legacy boxes keep byte-identical behaviour; running-only backfill; `backup.AppStopGuard` covers every stop→work→start window in its own marker file. **The durable fix for R-157 mechanism B.** Both blocking facts answered at source first: the crash-safe journal pattern DID already exist (quiesce, migrate) but covered **none** of the app-data path, and the SQLite store was rejected as a home because `metrics.db` is optional by design. `07`/`02` architecture docs carry the desired/in-flight/observed split. **Left for R-170:** the drive-backed boot gate still uses the container-count inference | -| R-170 | **The drive-backed boot gate still infers a Stop from a container count** — R-157 mechanism B, one gate over | S | READY — NEW 2026-08-02 | `shouldRecreateOnBoot` (`internal/web/intermediary.go:131`) ends in `&& hasContainers`. Added deliberately by **R-55** and correct for the signal available then; R-166 replaced the same inference in `bootrecon` and left this path behind. Fix: read `desired_state` with the legacy fallback, i.e. the same §8.1 table. Bounded consequence — `bootrecon` recovers the app seconds later, so what is lost is the gate's recreate-against-the-rebound-drive step, not the start | | R-158 | **A local Tier-1 backup failure reaches no hub channel** — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, only the off-box leg calls it | S | READY — 2026-08-02 | Fifth *seam built but never wired*; **R-97's defect one tier over**. Only caller `cmd/controller/main.go:659`; the manager's seams are tier2/offbox/offbox-enlarge (`backup.go:33,37,58`), none for the unit capture. **Ranked below R-157 by measurement:** `/backups` DOES show `✗ Adatmentés sikertelen`, it persists across a second failure and clears on recovery, and `/backups/apps` shows the last good unit's true mtime — so it is a notification gap, not silent failure. **The specific fix: `/backups/apps` is the page you would check for one app, and the only one that never says.** Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md`, `audits/CAMPAIGN-10-closeout-2026-08-02.md` | | R-159 | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy** | XS | **SHIPPED** (`templates/wishlist/`, 2026-08-02) — filed for the CLASS | Image declares `VOLUME /usr/src/app/data`; template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns names only for compose-declared volumes, so `DumpAppVolumes` never sees an anonymous one. **The class is open:** any image `VOLUME` at an unmounted path is silent unbacked-up storage — **`immich-server` has one today** at `/data`, empty when measured. Proposed `REUSE.md` rule: a template mounts every path in `Config.Volumes`, or says why not | | R-160 | **gramps-web persisted three paths and wrote to none of them** | XS | **SHIPPED** (`templates/gramps-web/`, 2026-08-02) | `/app/data` appears nowhere in the image's env; the accounts DB and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths, the template three, one a phantom. **Severity above papra's:** papra loses documents the customer may hold elsewhere; gramps-web loses the artefact built inside the app, of which no other copy exists by construction |