docs: R-55 PROVEN LIVE, new R-57 (app lifecycle), R-56 plumbing note, capability map

R-55's reboot leg ran operator-present on 9201: immich UI-stopped -> stayed
stopped across pct reboot, calibre-web recreated, zero alerts, ~15s.

R-57 records the lifecycle mechanism with the operator's abandoned-app
requirements verbatim and plant-it as the motivating case, including why the
retired/ directory move was wrong and the v0.158.1 pointer-receiver defect.

documentation/architecture/00-capability-map.md

@@ -62,8 +62,9 @@
 | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
 | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill |
 | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending |
-| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too** — with one honest caveat on the evidence: the fix is deployed and verified present in the running binary, but the reboot leg proving it end to end has NOT been run yet (it needs an operator-present `pct reboot 9201`), so this half is **shipped-not-yet-proven-live**. The **static-guest** half of the network leg stays deliberately out of scope → **R-50** |
+| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21** (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (`compose down` → 0 containers), calibre-web and bookstack left running, then `pct reboot 9201`: the gate recreated calibre-web and logged `1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`; immich came back **stopped**, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The **static-guest** half of the network leg stays deliberately out of scope → **R-50** |
 | Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) |
+| An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 |
 | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent, controller, bootstrap | **PARTIAL** | `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — a real relocation of the demo box: guest + hub telemetry + WG/PBS + Cloudflare tunnel all survived untouched, but the **controller↔agent control plane did not** (agent binds a LAN literal → `bind: cannot assign requested address` → storage/PBS-backup/quiesce/restore-test/DR down until fixed). Mitigated for the window by pinning `vmbr0` static | **R-50** (island-bridge control plane, spike-first) is the durable fix. Related: **R-51** (dead-primary alerting) and **R-52** (boot desired-state reconciliation) — the same event left two apps `Exited` with no alarm and no recovery |
 | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
 | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count |