docs: R-166 SHIPPED — the desired/in-flight/observed split (D-b)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
Controller v0.189.0 implements operator decision D-b. Docs only here; no hub change and no hub version bump. - architecture/02-controller-module-map.md §0a — NEW, and it is the S-1 contract: desired (app.yaml) / in-flight (own marker file) / observed (not persisted), with the rule that ties them — never derive one from another. Absent desired state means UNKNOWN, never "running". One file, one writer. D-b's binding safety rule quoted verbatim. - 00-capability-map.md — the boot-recovery row now rests on a recorded signal, with the three live flows from 9201. The interrupted-operation half is marked IMPLEMENTED, not PROVEN-LIVE: nobody killed the controller mid-backup on metal. - OPEN-ITEMS/ROADMAP — R-166 SHIPPED with both blocking facts and their answers; R-157 mechanism B CLOSED and A restated as the whole item; R-170 NEW (the drive-backed boot gate still infers a Stop from a container count). - STATUS.md — the "an app can stay switched off and nothing says so" line rewritten to what is actually left: timing. - CLAUDE.md — end-of-session checklist gains: confirm your own last push's CI run went green, BY RUN ID. The failure email is a push signal; this is the pull check. - CONTEXT.md S-12 — the rulings, and the two lessons worth carrying: a test that constructs the thing it should prove the caller constructs is hollow (its red-proof will say so), and a field-by-field struct rebuild in a save path is a defect on sight.
This commit is contained in:
@@ -122,6 +122,11 @@ pushes; **you (Claude Code) implement**. A file being open in the editor is NOT
|
||||
plain language, and is deliberately **not** `CONTEXT.md` — do not consolidate the two.
|
||||
- **A finding goes in `OPEN-ITEMS.md` first**, never only in a report, an audit or `STATUS.md`. Four
|
||||
items in this project were minted in a spike doc and lost (R-153/154/155, R-156/157).
|
||||
- **Confirm your own last push's CI run went green, by run ID.** CI emails on failure, which is a
|
||||
PUSH signal — this is the PULL check that catches a lost, filtered or unread mail. Quote the run
|
||||
id and its conclusion in the session report, e.g.
|
||||
`curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/<repo>/actions/tasks?limit=3"` → match the
|
||||
`head_sha` to your commit. An unchecked green is an assumption, not an observation.
|
||||
|
||||
## Tech stack (Hub)
|
||||
|
||||
|
||||
+43
@@ -164,6 +164,49 @@ Five decisions were deliberately **left open for the operator** and are recorded
|
||||
stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with
|
||||
the guest it backs up. Gaps minted the same session: **R-102 … R-108**.
|
||||
|
||||
**S-12 — D-b is BUILT (controller v0.189.0, 2026-08-02, R-166).** The desired/in-flight/observed
|
||||
split now exists; the S-1 contract lives in `architecture/02-controller-module-map.md` §0a.
|
||||
|
||||
**Both facts D-b said to establish first were established at source, and both changed the shape.**
|
||||
(a) *Does the crash-safe journal in the backup code already cover the in-flight case?* The pattern
|
||||
DID already exist — twice (`quiesce` marker+`Recover`, `migrate` journal+`RecoverMigration`) — and
|
||||
covered **none** of the app-data path: `DumpAppVolumesSafe` stopped and restarted an app with no
|
||||
marker, no journal and **not even a `defer`**. So the answer was neither "it exists, wire it" nor
|
||||
"build it": copy the proven shape into its own file. It was the fifth time the question was worth
|
||||
asking and the first time the answer was "the pattern, not the coverage". (b) *Is the SQLite store
|
||||
reachable?* Reachable and **deliberately not used** — `metrics.db` is optional by design (the
|
||||
controller runs with it absent), and operational state must not live in a store built to be dropped.
|
||||
|
||||
**The ruling that carried the design: absent means UNKNOWN, never "running".** Every `app.yaml` on
|
||||
every box predates the field, so absent is what the whole fleet reads on upgrade; reading it as
|
||||
running would have started every deliberately-stopped app on the first boot after the upgrade —
|
||||
fleet-wide, silently. Where intent is unknown the box keeps the OLD inference rather than inventing
|
||||
an answer. That is also why the backfill is **running-only**: "zero containers ⇒ stopped" is the
|
||||
defect itself, so an ambiguous app stays ambiguous until a customer presses a button.
|
||||
|
||||
**The other load-bearing ruling: `StartStack`/`StopStack` are NOT writers of intent.** A census found
|
||||
14 callers, of which exactly 2 are the customer. Recording intent in the primitive would make a
|
||||
nightly backup indistinguishable from the customer pressing Stop — the confusion being removed.
|
||||
|
||||
**Found on the way, and it would have silently eaten the feature: `SaveAppConfig` rebuilt `AppConfig`
|
||||
field-by-field.** That is the R-100 shape, which v0.181.0 shipped two live instances of. The literal
|
||||
named five fields, so the sixth would have been dropped on every save across nine call sites — a
|
||||
customer's Stop erased by the next unrelated `app.yaml` write. Copy-and-overlay is safe by
|
||||
construction; the failure mode is generic, so **treat any field-by-field struct rebuild in a save
|
||||
path as a defect on sight.** Measured and documented rather than assumed: `app.yaml` does NOT
|
||||
round-trip YAML keys the struct does not model.
|
||||
|
||||
**Closes R-157 mechanism B; mechanism A (the sweep observes ~5 s after start and never re-checks) is
|
||||
untouched and is now the whole of R-157** — and B's fix makes A cost more, since the sweep now has
|
||||
more it could legitimately recover. **New: R-170** — `shouldRecreateOnBoot`
|
||||
(`internal/web/intermediary.go:131`) still infers a Stop from `hasContainers`, i.e. the same defect
|
||||
one gate over for drive-backed apps. Left deliberately: the task scoped `bootrecon`, and two boot
|
||||
behaviour changes under one live validation is one too many.
|
||||
|
||||
**Live on 9201, three flows, each with a positive observable.** The interrupted-operation half is
|
||||
**IMPLEMENTED, not PROVEN-LIVE** — unit-proven and red-proofed, but nobody killed the controller
|
||||
mid-backup on real hardware; the capability map says so rather than rounding it up.
|
||||
|
||||
**S-5 — four operator decisions taken in discussion on 2026-08-02, recorded before anything is
|
||||
built.** They existed only in conversation, which is the condition the standing rules were written
|
||||
against. Labels are the ones used in the discussion (**D-a … D-d**) and are deliberately kept
|
||||
|
||||
@@ -18,9 +18,13 @@ restore files and app data from the drive alone. Proven end to end on real hardw
|
||||
|
||||
## What's broken
|
||||
|
||||
**After a power cut, an app can stay switched off — and nothing says so.** The machine looks for apps
|
||||
that didn't come back, but looks too early and never again. In one case it decides the customer
|
||||
switched it off deliberately, so it isn't even counted as down. *(R-157)*
|
||||
**After a power cut, an app can stay switched off — and the machine may look too early to notice.**
|
||||
It now knows the difference between an app the customer switched off and one that simply didn't come
|
||||
back: the customer's choice is written down, so an app meant to be running is started again however
|
||||
it went missing. What is left is timing — it looks about five seconds after starting, while the
|
||||
system is still coming up, and never looks again. *(R-157; the "decided the customer switched it off"
|
||||
half is fixed — R-166, 2026-08-02. The same guess still lives in one other place, for apps kept on a
|
||||
plug-in drive — R-170.)*
|
||||
|
||||
**The off-site copy can be erased by the machine that made it** — the credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
|
||||
@@ -80,7 +80,7 @@
|
||||
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
|
||||
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill **Route + RTO → `07-backup-architecture.md` §8 rows 6, 8, 9** — measured 84–112 s local / 1101 s PBS into a scratch guest; a restore to a DIFFERENT host is unmeasured |
|
||||
| PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending |
|
||||
| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21** (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (`compose down` → 0 containers), calibre-web and bookstack left running, then `pct reboot 9201`: the gate recreated calibre-web and logged `1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`; immich came back **stopped**, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The **static-guest** half of the network leg stays deliberately out of scope → **R-50** |
|
||||
| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→**v0.189.0**, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21; boot-orphan leg re-proven on the new signal 2026-08-02) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21** (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (`compose down` → 0 containers), calibre-web and bookstack left running, then `pct reboot 9201`: the gate recreated calibre-web and logged `1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`; immich came back **stopped**, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The **static-guest** half of the network leg stays deliberately out of scope → **R-50**. **2026-08-02 — the boot-orphan leg now rests on a RECORDED signal, not an inference (R-166, controller v0.189.0).** Both the R-52 sweep and the R-55 gate above decided "the customer stopped this" from *zero containers*, which is also what a power cut mid-compose and an interrupted deploy leave behind — so two real faults were read as deliberate stops and stranded silently (R-157 mechanism B). The customer's intent is now written to `app.yaml` (`desired_state`) by their own action and read directly. **PROVEN-LIVE on 9201** in three flows: a UI Stop persisted `stopped` and survived a controller restart with the app still down and NOT listed as a candidate; an app recorded `running` whose containers were removed out-of-band was recovered by name (`[bootrecon] 1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt) — **the case that was invisible before**; and a legacy `app.yaml` with no field was skipped exactly as before and was never inferred to be stopped. **Interrupted app-data operations are covered separately and are NOT proven-live** — `backup.AppStopGuard` restarts apps left stopped by a killed volume dump / offsite reconstitute / `.fab` export, and that leg is unit-proven + red-proofed only (killing the controller mid-backup on a live box was not exercised): **IMPLEMENTED, not PROVEN-LIVE.** **R-55's own gate was NOT migrated → R-170** |
|
||||
| Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) |
|
||||
| An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 |
|
||||
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
|
||||
|
||||
@@ -53,6 +53,80 @@ Risk tags: **clean** · **needs-rework** · **hazard** (entangles a delete-targe
|
||||
|
||||
---
|
||||
|
||||
## 0a. App state: desired / in-flight / observed — S-1 CONTRACT (2026-08-02, decision D-b, R-166)
|
||||
|
||||
> **This section is a live contract, not migration history** — the rest of this document is the
|
||||
> v0.33 keep/port/delete inventory. Read this before touching `stacks/`, `backup/` or `bootrecon/`.
|
||||
> Shipped in controller **v0.189.0**.
|
||||
|
||||
An app's state is **three different kinds of information**, and conflating them is what produced
|
||||
R-157 mechanism B and F-CRIT-1. They are stored differently on purpose.
|
||||
|
||||
| Kind | Question it answers | Where it lives | Persisted? |
|
||||
|---|---|---|---|
|
||||
| **Desired** | *What did the customer ask for?* | `app.yaml` → `desired_state` | Yes, beside the app's other settings |
|
||||
| **In-flight** | *Is an operation part-way through, and did it finish?* | its **own** marker file under `<data_dir>` | Yes, written before the operation and cleared after |
|
||||
| **Observed** | *Is it running, unhealthy, restarting, is its drive gone?* | nowhere | **No — rebuilt by looking** |
|
||||
|
||||
**The rule that ties them together: never derive one from another.** The defect this replaced did
|
||||
exactly that — it derived *desired* from *observed* (zero containers ⇒ "the customer stopped it"),
|
||||
and zero containers is equally what a power cut mid-compose, an interrupted deploy and an interrupted
|
||||
backup leave behind. Two real faults were therefore read as deliberate stops and stranded silently.
|
||||
|
||||
### Desired — `app.yaml`, `desired_state`
|
||||
|
||||
Tri-state: `""` (unknown) · `"running"` · `"stopped"`.
|
||||
|
||||
- **ONE OWNER: the customer's own action.** Writers are the `/api/stacks/{name}/{action}` switch,
|
||||
`DeployStack`, `UpdateOptionalConfig`'s redeploy branch, and the `.fab` import. **`StartStack` and
|
||||
`StopStack` are NOT writers** — a census found 14 callers of which only 2 are the customer; the
|
||||
rest are quiesce, the backup volume dump, offbox reconstitution, app export/restore, the storage
|
||||
gate, migration and the boot reconciler. Intent recorded in the primitive would make a nightly
|
||||
backup indistinguishable from the customer pressing Stop.
|
||||
- **Written BEFORE the act; a failed write REFUSES the act.**
|
||||
- **Absent means UNKNOWN — never "running".** Every `app.yaml` predating v0.189.0 lacks the field,
|
||||
so consumers must fall back to the pre-v0.189.0 behaviour rather than assume. A running-only
|
||||
backfill converges the unambiguous cases; **`stopped` is never inferred, from any signal.**
|
||||
|
||||
### In-flight — a marker file, one per owner
|
||||
|
||||
Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
|
||||
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
|
||||
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
|
||||
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
|
||||
note is a stranded app by a different route.
|
||||
|
||||
- Written **before** the stop; cleared **only** after a restart that succeeded; a **failed** restart
|
||||
keeps the marker so the next startup retries.
|
||||
- **A `defer` is not the mechanism.** A SIGKILL runs no deferred function — established on live
|
||||
hardware by Campaign 8 fault 10, where what brought the stacks back was the marker read at startup.
|
||||
- Recovery runs at startup and **completes before** the boot reconciler is launched, so an app the
|
||||
marker explains is not also reported as an unexplained boot orphan.
|
||||
|
||||
### Observed — not persisted, by design
|
||||
|
||||
`aggregateState` walks **every container** of a stack and any unhealthy or mixed result wins, so a
|
||||
partly-dead app cannot read as healthy (F-CRIT-1's shape). This requirement is met here and must not
|
||||
be re-implemented downstream. Nothing about observed state is written to disk: a controller restart
|
||||
re-observes it within one refresh, whereas persisting it risks carrying a stale verdict across the
|
||||
very restart that fixed it (the same argument as `RestartingSince`).
|
||||
|
||||
### The binding safety rule (verbatim, from decision D-b)
|
||||
|
||||
> *Losing the state store must never cause an app to be deleted, restarted wrongly, or reported
|
||||
> healthy when it is not — the worst acceptable outcome is re-running a backup that already ran.*
|
||||
|
||||
Applied: a lost or corrupt marker means the app is not auto-restarted **by that mechanism**, which is
|
||||
the pre-v0.189.0 position, not a new hazard. A lost `app.yaml` already means the app is not deployed.
|
||||
**Nothing here may make an absent file more dangerous than a present one.**
|
||||
|
||||
### Not migrated
|
||||
|
||||
`shouldRecreateOnBoot` (`internal/web/intermediary.go`) — the drive-backed boot gate — still derives
|
||||
desired from observed via `hasContainers`. Tracked as **R-170**.
|
||||
|
||||
---
|
||||
|
||||
## 1. v0.33 module inventory (package → purpose, key deps)
|
||||
|
||||
| Package | Purpose | Key internal deps |
|
||||
|
||||
@@ -85,7 +85,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-138** | **A shared-zone `cf_api_token` is a zone-wide DNS-write capability on a customer's box** — written 0600 to `/opt/docker/stacks/traefik/.env` (`controller/internal/infra/infra.go:123`) | READY (S) | — | Today each box holds a token for a zone nobody else uses, so the blast radius is one customer. Under a shared customer zone, one compromised tester box could repoint every other tester's DNS. The ACME path is already switchable — an empty token selects HTTP-01 (`traefik.yml.tmpl`) — so the fix is policy plus a guard that refuses to hand a shared-zone customer a zone-scoped token. Same audit §5.2 | CC |
|
||||
| **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC |
|
||||
| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. **THE REFERRAL IS RESOLVED, 2026-08-02 — papra is deployed NOWHERE, so the template fix strands nothing and can be applied.** The referral existed because changing where the volume mounts moves live data: an installed papra writes `db.sqlite` into the container's writable layer, and a remount relocates the path out from under it. With no instance deployed there is no live data to move, so the cheaper leg — **the template mounts `/app/app-data`** — is takeable directly, without waiting on upstream to adopt `/app/data`. **Provenance, stated because it decides the row:** the observation is `docker ps -a` on demo-hp's **guest 9201** returning empty, supplied with the 2026-08-02 task; **this session did not re-measure** (documentation-only, every box fenced). **Scope of that evidence, honestly:** it covers guest 9201 — the guest papra was convicted on in Campaign 10 — and **no other customer's guest was enumerated**, so a re-check belongs in the task that edits the template, before it edits it. **Next action: apply the template fix in `app-catalog-felhom.eu` and re-run `scripts/catalog_gates.py`** (deliberately not done here — that repo was out of scope for this task). See R-161 (nothing runs the gate automatically) and R-159/R-160 | CC |
|
||||
| **R-157** | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms, and a settle-condition fix closes only one.** **(A)** the sweep runs ~5 s after controller start (`cmd/controller/main.go:236`) while docker is still restoring, sees "no boot-orphaned apps", and never re-checks — **intermittent, 3 of 6 hard resets**; the predicate itself is sound (`bootrecon.go:100`, and `IsDownState` includes `StateDegraded`, `manager.go:55`). **(B)** an interruption that lands mid-compose leaves the stack with **zero containers**, which is exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp check reported **`0 currently down`** while a `deployed: true` app was not running: **silent on every channel**. | **READY (M)** | — | **Two fixes, not one.** A: a settle condition before the single observation, or one re-evaluation at the end of `deadAppBootGrace`, still bounded to the same attempts. B: needs a signature that distinguishes "customer stopped this" from "interrupted between containers" — B is F-CRIT-1's shape (`CLAUDE.md` false-invariant #4) reached via hard reset rather than quiesce. Test the **consequence** (app running again before the grace expires), not the mechanism. Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` | CC |
|
||||
| **R-157** | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms. MECHANISM B IS CLOSED; MECHANISM A REMAINS.** **(A) STILL OPEN:** the sweep runs ~5 s after controller start (`cmd/controller/main.go`, `runBootReconcile`) while docker is still restoring, sees "no boot-orphaned apps", and never re-checks — **intermittent, 3 of 6 hard resets**. R-166 did **not** touch the timing, so this is untouched and is now the whole of R-157. **(B) CLOSED by R-166 (controller v0.189.0, 2026-08-02):** an interruption mid-compose left the stack with **zero containers**, the signature bootrecon deliberately skipped as a user's Stop. The sweep now reads the customer's RECORDED intent (`desired_state` in `app.yaml`), so `running` + zero containers is an orphan and is recovered — **proven live on 9201**: containers removed out-of-band, controller restarted, `[bootrecon] 1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt. | **READY (S) — A only** | — | **A is now a pure timing fix and is the only thing left here:** a settle condition before the single observation, or one re-evaluation at the end of `deadAppBootGrace`, still bounded to the same attempts. Test the **consequence** (app running again before the grace expires), not the mechanism. Note that A and B were independent: B was a wrong signal, A is a premature observation, and B's fix makes A **more** visible (the sweep now has more it could recover, so observing too early costs more). Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`; B's closure in `felhom-controller/REPORT.md` (2026-08-02) | CC |
|
||||
| **R-170** | **The drive-backed boot gate still infers a customer's Stop from a container count — R-157 mechanism B, one gate over, for drive-backed apps only.** `shouldRecreateOnBoot` (`controller/internal/web/intermediary.go:131`) ends in `&& hasContainers`, sampled at `:425` as `len(st.Containers) > 0`. That term was added deliberately by **R-55** (controller v0.157.0) to stop the gate resurrecting an app the customer had stopped, and it was the right fix with the signal available then. R-166 replaced exactly that inference in `bootrecon` with the recorded `desired_state`, and **this gate was left on the old signal** — so for a drive-backed app, a power cut or interrupted deploy that leaves zero containers is still read as a deliberate stop by this path | **READY (S) — NEW 2026-08-02** | — | **NOT acted on by R-166, deliberately** — the task scoped `bootrecon` and named this gate nowhere, and changing a second boot path in the same release would have shipped two behaviour changes under one live validation. **Consequence is bounded, not zero:** `bootrecon` now recovers the same app a few seconds later on intent, so the observable defect is narrowed to the gate's own recreate-onto-the-drive step being skipped — which matters when the app needs recreating against a re-bound drive, not merely starting. **Fix shape:** replace `hasContainers` with `stacks.DesiredStateOf(st) != DesiredStateStopped` and keep the legacy fallback, i.e. the §8.1 table this gate already half-implements. **Established free:** `grep -ro "R-170\b" documentation/ *.md` → 0 hits (2026-08-02) | CC |
|
||||
| **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC |
|
||||
| **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `<project>_<name>` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC |
|
||||
| **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |
|
||||
@@ -94,7 +95,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio** | — (the sizing question is answered; the work is **R-165**) | **RE-FRAMED, NOT CLOSED (operator decision D-a, 2026-08-02 — `CONTEXT.md` S-5).** The row asked *what ratio should `mp1` be?* and that question is **withdrawn rather than answered**: `mp1` is merged into `mp0` so local recovery units share the app-data area and the ceiling stops existing — a bigger number is the same wall further away. **This row stays open as the record of the constraint** (what `mp1` is for, what it gates, and the measured 2× DB-app unit size) **until the merge lands**, because until then every consequence below is still live on every box. **The work is R-165; the warning that must ship with it is R-167.** Original finding, unchanged, follows. **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 (widened to R-167) | CC |
|
||||
| **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
|
||||
| **R-165** | **Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.** Operator decision **D-a**, 2026-08-02 (`CONTEXT.md` S-5). Local recovery units share the app-data area instead of holding their own fixed ceiling, so the wall R-163 describes is removed rather than moved further away. Guest 9201 on demo-hp ships `mp0 50G / mp1 20G` today | **READY (M) — NEW 2026-08-02** | — | **Two conditions travel WITH the decision and are not optional.** **(1) Before any external install.** It changes the **disk layout**, so it is a fresh-install shape while there are no external boxes and a per-box migration after — and the decision's cheapness is entirely a function of that ordering. **(2) It removes a wall that currently fails safely**, so **R-167** (D-c: fill warning + failure alert) lands in the same step, never after: today an app that outgrows `mp1` is refused per app with the last good unit preserved byte-identical (R-158's measurement), and after the merge the same overflow consumes the space the app itself is using. Touches the installer/agent guest shape (`--sysdata-grow` defaults to **0** and is not derived from the physical drive, `felhom-agent/cmd/felhom-agent/main.go:178`) and the golden. **Does NOT close R-163** — that row is the record of the constraint and stays open until this lands | CC |
|
||||
| **R-166** | **App state gets a desired/observed model with its own store.** Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5). Three kinds of information, deliberately separated: **desired** (running, or stopped because the customer said so) stays in `app.yaml`; **in-flight operations** (a backup started and never reported finishing) are **persisted** — written before the operation and after it — so an interruption is visible after a reboot; **observed** (unhealthy, restarting, drive gone) is **not** persisted and is rebuilt by looking. The last two live in a store **separate from `app.yaml`**, so frequent writes never touch a file holding deploy settings and secrets | **BLOCKED — NEW 2026-08-02** | two facts to establish before it is specced | **NOT a spec yet, deliberately.** **Establish first:** (a) whether the crash-safe journal already in the backup code covers the in-flight case — *it may already exist and merely be unwired, which has been the answer four times in this project* (`CLAUDE.md` "seam built but never wired"); (b) whether the existing SQLite store is reachable from this code path. **Binding safety rule, verbatim from the decision:** *losing the state store must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the worst acceptable outcome is re-running a backup that already ran.* The controller must also check **every container of an app**, so a partly-dead app cannot read as healthy. **This is the durable fix for R-157 mechanism B** (zero containers reading as a deliberate stop) and for F-CRIT-1's class; R-157 keeps its own row because it wants a bounded fix now, not this rework | CC |
|
||||
| **R-166** | ~~**App state gets a desired/observed model with its own store.**~~ Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5) | **SHIPPED + PROVEN-LIVE** (controller v0.189.0, 2026-08-02) | — | **Both blocking facts were established at source before any code was written, and the answers changed the shape.** **(a) Does a crash-safe journal already exist for the in-flight case?** YES, twice — `internal/quiesce/quiesce.go` (marker + `Recover`, proven on live hardware by Campaign 8 fault 10) and `internal/stacks/migrate.go` (`migration.json` + `RecoverMigration`) — but **neither covers the app-data path**: `DumpAppVolumesSafe` stopped and restarted an app with **no marker, no journal and not even a `defer`**. So the pattern existed and the coverage did not; `backup.AppStopGuard` copies the proven shape into its **own** file (one file, one writer). **(b) Is the SQLite store reachable?** Irrelevant, and deliberately unused: `metrics.db` is optional by design (the controller runs with it absent), and operational state must not live in a store designed to be droppable. **Shipped:** tri-state `desired_state` in `app.yaml` written ONLY by the customer's action (API action switch, `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, `.fab` import — a 14-caller census established that `StartStack`/`StopStack` must NOT be writers); `isBootOrphan` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN, never running**, so a legacy `app.yaml` keeps byte-identical pre-v0.189.0 behaviour; running-only backfill. D-b's every-container requirement was already met by `aggregateState` and was NOT re-implemented. **Live on 9201:** all three flows (stop survives a restart; a zero-container `running` app is recovered by name; a legacy app.yaml is skipped and never inferred as stopped). **Also fixed en route:** `SaveAppConfig` rebuilt `AppConfig` field-by-field (the R-100 shape) and would have dropped the new field on every save across nine call sites | — |
|
||||
| **R-167** | **Storage monitoring and backup alerts.** Operator decision **D-c**, 2026-08-02 (`CONTEXT.md` S-5). Two pieces: a warning to the **customer** as a drive fills, **before** anything fails; and a failure alert to the **hub/operator** when any backup fails for **any** reason, lack of space included | **READY (M) — NEW 2026-08-02** | — | **Prerequisite for R-165** — D-a removes a wall that currently fails safely, so this ships in the same step, not after. **Subsumes and widens R-158**, which is the same alert one tier down (`NotifyBackupFailed` exists and the hub allowlists `backup_failed`; only the off-box/NAS leg calls it, `cmd/controller/main.go:659`). R-158 stays as the named seam and its proposed `unitNotify` shape is the concrete first leg — **do not file a second row for the same wire.** The customer half is new: nothing today warns before a fill, and R-158's measurement is that when it happens exactly one page says so | CC |
|
||||
| **R-168** | ~~CI: no runner exists, and with trunk-based pushes CI can DETECT but not BLOCK~~ | **SHIPPED — and the alarm is DEMONSTRATED** (2026-08-02) | — | **Runner live**: `homelab-manifests/gitea-system/act-runner.yaml`, an unprivileged host-mode `act_runner` in `gitea-system`, one owner-scoped registration serving all four repos (measured: tasks 7-10 all claimed by `felhom-gates-runner`). `.gitea/workflows/gates.yml` in each repo runs that repo's entry point with `--fast` and nothing else; no `uses:` step anywhere. **Six probes, all answered, none STOPped** — `audits/SPIKE-ci-runner-2026-08-02.md`. The two that changed the design: **P2** (stock image has git but NO python3 → custom image `felhom-act-runner:0.1.0`, base pinned, python3 and nothing else) and **P6** (a runner that loses `/data/.runner` re-registers and leaves a dead record behind → the PVC is load-bearing, measured both ways). **P5 is the one that mattered**: a failed run produced NO mail, NO notification row and NO log line from Gitea, so the run now sends its own alarm via Resend and prints the provider's accepted id. **Proven end to end, not asserted**: a deliberately broken commit pushed with `--no-verify` → run #6 `failure` → `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. Posture shown from the live pod spec: `privileged: false`, all caps dropped, no docker socket, no hostPath, `automountServiceAccountToken: false`, sized at half Gitea's limits so it cannot crowd out the service holding every repository on the same node. **The standing limit stays true and is written into the manifest and every workflow: it DETECTS, it does not BLOCK** — making it block is → R-169 | — |
|
||||
| **R-169** | **CI can only report, because there is no gate in the road.** Every felhom repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand at. R-168's runner therefore notices a broken push *after* it has landed | **WAITING-ON-OPERATOR** (a working-style decision, not a defect) | an operator ruling | Making CI *blocking* requires two things this task deliberately did NOT do, because both change how the operator works and that is not a task's call: **(a)** branch protection on `main`, and **(b)** a pull-request workflow instead of direct-to-`main` pushes. The cost is real — every change would need a PR, which for a single-operator project may be worse than the disease. **The current arrangement is two nets, and it is not nothing**: `.githooks/pre-push` REFUSES locally, and R-168's runner NOTICES when that hook was skipped or was never armed in a clone, and emails. The honest gap is the window between a `--no-verify` push landing and the operator reading the alarm. Decide only if that window ever actually costs something | operator |
|
||||
|
||||
@@ -162,7 +162,9 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
|
||||
| R-154 | **`[first-boot]` is automated-install-only, and nothing in the tree said so** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | A PVE property, measured with a same-image control (`audits/SPIKE-universal-iso-3-2026-07-31.md` §2); recorded at `scripts/iso/pkg/build-deb.sh:6-11`. Superseded in practice by the `.deb` delivery route |
|
||||
| R-155 | **`iso-repack.sh` refused any ISO without `auto-installer-mode.toml`** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | **Narrowed, not deleted** — unchanged for `FELHOM_MENU=single` (`iso-repack.sh:121-128`), does not apply to `release` where the file's absence *is* gate G1. Do not remove it wholesale |
|
||||
| R-156 | **An app's data is neither persisted nor backed up, and it reports healthy** — a template mounts a volume at a path the app never writes, so data sits in the container's writable layer: lost on redeploy, tarred nightly as an empty dir, healthcheck green | S | **DETECTED + GATE SHIPPED; papra REFERRED** (catalog sweep, 2026-08-02) | Found by Campaign 10 on **papra**; the 53-template sweep convicted **gramps-web** and **wishlist** too and **fixed both**. The gate is `app-catalog-felhom.eu/scripts/check-volume-persistence.py` — a runtime probe (`docker diff` + mount occupancy + writability, canary self-test, fails closed); the detector and the gate are one program. **papra is NOT fixed**: the fix needs the app to use `/app/data` or the template to mount `/app/app-data`, so it is referred. Sweep evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` (53 probe.json). Ranking in `OPEN-ITEMS.md` |
|
||||
| R-157 | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — two mechanisms** | M | READY — 2026-08-02 | R-52 built `internal/bootrecon` for exactly this (F5: two apps `Exited` ~18 h). **(A)** the sweep fires ~5 s after start (`main.go:236`) while docker is still restoring and never re-checks — **3 of 6 hard resets**, intermittent, which is what a race predicts and a wrong predicate does not; the predicate is sound (`bootrecon.go:100`; `IsDownState` includes `StateDegraded`, `manager.go:55`). **(B)** a mid-compose interruption leaves **zero containers**, the signature bootrecon deliberately skips as a user Stop — and deadapp then reported **`0 currently down`** for a `deployed: true` app that was not running. **A settle-condition fix closes A and leaves B open.** Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` |
|
||||
| R-157 | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — two mechanisms** | S | **B SHIPPED (controller v0.189.0, 2026-08-02) — A still READY** | R-52 built `internal/bootrecon` for exactly this (F5: two apps `Exited` ~18 h). **(B) CLOSED by R-166:** a mid-compose interruption left **zero containers**, the signature bootrecon skipped as a user Stop; the sweep now reads the recorded `desired_state` instead of counting containers, so `running` + zero containers is an orphan — **proven live on 9201** (containers removed out-of-band → `1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt). **(A) UNTOUCHED and now the whole item:** the sweep fires ~5 s after start while docker is still restoring and never re-checks — **3 of 6 hard resets**, intermittent, which is what a race predicts and a wrong predicate does not. R-166 deliberately did not change the timing. **B's fix makes A cost more**, since the sweep now has more it could legitimately recover. Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` |
|
||||
| R-166 | **App state gets a desired/observed model with its own store** (operator decision D-b) | M | **SHIPPED + PROVEN-LIVE — controller v0.189.0, 2026-08-02** | Tri-state `desired_state` in `app.yaml`, written ONLY by the customer's action; `bootrecon` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN** so legacy boxes keep byte-identical behaviour; running-only backfill; `backup.AppStopGuard` covers every stop→work→start window in its own marker file. **The durable fix for R-157 mechanism B.** Both blocking facts answered at source first: the crash-safe journal pattern DID already exist (quiesce, migrate) but covered **none** of the app-data path, and the SQLite store was rejected as a home because `metrics.db` is optional by design. `07`/`02` architecture docs carry the desired/in-flight/observed split. **Left for R-170:** the drive-backed boot gate still uses the container-count inference |
|
||||
| R-170 | **The drive-backed boot gate still infers a Stop from a container count** — R-157 mechanism B, one gate over | S | READY — NEW 2026-08-02 | `shouldRecreateOnBoot` (`internal/web/intermediary.go:131`) ends in `&& hasContainers`. Added deliberately by **R-55** and correct for the signal available then; R-166 replaced the same inference in `bootrecon` and left this path behind. Fix: read `desired_state` with the legacy fallback, i.e. the same §8.1 table. Bounded consequence — `bootrecon` recovers the app seconds later, so what is lost is the gate's recreate-against-the-rebound-drive step, not the start |
|
||||
| R-158 | **A local Tier-1 backup failure reaches no hub channel** — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, only the off-box leg calls it | S | READY — 2026-08-02 | Fifth *seam built but never wired*; **R-97's defect one tier over**. Only caller `cmd/controller/main.go:659`; the manager's seams are tier2/offbox/offbox-enlarge (`backup.go:33,37,58`), none for the unit capture. **Ranked below R-157 by measurement:** `/backups` DOES show `✗ Adatmentés sikertelen`, it persists across a second failure and clears on recovery, and `/backups/apps` shows the last good unit's true mtime — so it is a notification gap, not silent failure. **The specific fix: `/backups/apps` is the page you would check for one app, and the only one that never says.** Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md`, `audits/CAMPAIGN-10-closeout-2026-08-02.md` |
|
||||
| R-159 | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy** | XS | **SHIPPED** (`templates/wishlist/`, 2026-08-02) — filed for the CLASS | Image declares `VOLUME /usr/src/app/data`; template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns names only for compose-declared volumes, so `DumpAppVolumes` never sees an anonymous one. **The class is open:** any image `VOLUME` at an unmounted path is silent unbacked-up storage — **`immich-server` has one today** at `/data`, empty when measured. Proposed `REUSE.md` rule: a template mounts every path in `Config.Volumes`, or says why not |
|
||||
| R-160 | **gramps-web persisted three paths and wrote to none of them** | XS | **SHIPPED** (`templates/gramps-web/`, 2026-08-02) | `/app/data` appears nowhere in the image's env; the accounts DB and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths, the template three, one a phantom. **Severity above papra's:** papra loses documents the customer may hold elsewhere; gramps-web loses the artefact built inside the app, of which no other copy exists by construction |
|
||||
|
||||
Reference in New Issue
Block a user