Controller v0.190.0. Docs only here; no hub change, no hub version bump. - audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis, including the run that produced a FALSE NEGATIVE and the mechanism behind it (the agent re-binds an unmounted drive within ~60s, so the drive gate's startup reconcile restarted the apps one second before the sweep looked). Records that the write hazard was blocked only by an ACCIDENTAL filesystem permission that no code owns and no test pins. - architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired state; the sweep observes a SETTLED fleet and each sample must refresh first; nothing is started without asking, fail-safe. Plus the durable warning: Manager.StartStack has no gate of its own. - 00-capability-map — the boot-recovery row, with the repeat count cited per N.5 (6 of 6 hard resets) rather than a bare PROVEN-LIVE. - OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW and closed the same session, marked a regression from v0.189.0. - STATUS.md — the power-cut line moved from "What's broken" to "What works right now" with its repeat count; one dated bullet in the change log. - CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is not a disproof; widening a window makes previously-unreachable overlaps reachable; and a settle detector is only as good as the freshness of what it samples — the fix's own defect, found live rather than by review.
This commit is contained in:
+50
@@ -164,6 +164,56 @@ Five decisions were deliberately **left open for the operator** and are recorded
|
||||
stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with
|
||||
the guest it backs up. Gaps minted the same session: **R-102 … R-108**.
|
||||
|
||||
**S-13 — boot recovery finished, and the lesson is about the DIAGNOSIS ORDER (controller v0.190.0,
|
||||
2026-08-02, R-157 A · R-170 · R-171).**
|
||||
|
||||
**The session's most valuable half hour was spent NOT writing code.** A hole was reasoned out of the
|
||||
v0.189.0 diff — replacing the container-count term with recorded intent should make a
|
||||
drive-gate-stopped app read as a boot orphan — and the task's own rule was to CONFIRM it on hardware
|
||||
before writing a fix. **The first attempt to confirm it produced a false negative**, and reporting
|
||||
that as a disproof would have been wrong: unmounting only the parent bind is healed by the agent
|
||||
within ~60 s, so the drive gate's startup reconcile restarted the apps **one second before** the
|
||||
sweep looked. `no boot-orphaned apps` in that log is a race that went the safe way, not a mechanism.
|
||||
Holding the drive genuinely absent reproduced it immediately. **"It didn't happen this time" is not a
|
||||
disproof — name the mechanism or run it again.**
|
||||
|
||||
**The confirmation also changed the severity, in both directions.** The write hazard did NOT
|
||||
materialise: compose failed `mkdir …/userdata: permission denied`, because the unbound mountpoint is
|
||||
host-root-owned and the guest is unprivileged. **That protection is accidental** — no code chose it,
|
||||
no test pins it, it is one `chown` or one privileged guest away from gone, and its removal would be
|
||||
invisible until data landed on the wrong disk. Meanwhile the harm that DID occur was real on every
|
||||
box and was not in the hypothesis: two wasted attempts and a **false dead-app alarm for an app the
|
||||
drive gate is deliberately holding**. Diagnosing first is what produced both facts.
|
||||
|
||||
**The fix was already in the codebase, on another path.** The API's `startGatedByMissingDrive`
|
||||
refuses a customer's start on an absent drive with a Hungarian message. The sweep bypassed it by
|
||||
calling `Manager.StartStack` directly. **`StartStack` has no gate of its own** — that is the durable
|
||||
fact worth carrying: every caller that is not the customer must decide for itself whether the app may
|
||||
run, and there are now fourteen of them.
|
||||
|
||||
**Widening a window makes previously-unreachable overlaps reachable, and that is a design input, not
|
||||
an afterthought.** The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a
|
||||
50 s window can. All three holders answer one seam rather than three, because they differ only in
|
||||
the reason string.
|
||||
|
||||
**A test rejected my first constant, and the comment now says so.** `settle + budget + one retry`
|
||||
must fit inside `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s
|
||||
against a 90 s grace. The budget is 50 s **because a test said so**, and the code records that rather
|
||||
than presenting the number as taste. Widening the grace to fit was rejected outright: it hides a late
|
||||
recovery instead of reporting one.
|
||||
|
||||
**AND THE FIX HAD ITS OWN DEFECT, FOUND BY LIVE VALIDATION AND NOT BY REVIEW.** The window sampled
|
||||
`GetStacks()` — the Manager's in-memory map, refreshed by the scheduler every 10 s — every 5 s. Two
|
||||
identical samples could therefore mean *the cache did not update*, not that the fleet had settled. It
|
||||
surfaced as a container removed ~5 s before the window closed still being in the sampled fleet, with
|
||||
the sweep logging `no boot-orphaned apps` for an app that had none. **Generalise it: a settle
|
||||
detector is only as good as the freshness of what it samples — if the source is cached, refresh it or
|
||||
you are watching the cache settle, not the system.**
|
||||
|
||||
**Live: 6/6 hard resets on the shipped build** (every app back; a customer-stopped app down in all
|
||||
six), window settle times 10/40/10/10/15/15 s — routinely 2–8× the old fixed 5 s. The sharpest
|
||||
evidence is a same-app before/after on one box: missed at 18:08:35, recovered at 18:18:50.
|
||||
|
||||
**S-12 — D-b is BUILT (controller v0.189.0, 2026-08-02, R-166).** The desired/in-flight/observed
|
||||
split now exists; the S-1 contract lives in `architecture/02-controller-module-map.md` §0a.
|
||||
|
||||
|
||||
@@ -16,15 +16,12 @@ home network, and open apps from a launcher or a shared link. Backups run on the
|
||||
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
||||
restore files and app data from the drive alone. Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
**Apps come back after a power cut.** The machine tells an app the customer switched off from one
|
||||
that simply did not come back, and waits for the system to finish starting before deciding instead of
|
||||
glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every
|
||||
time, and an app switched off deliberately stayed off every time.
|
||||
|
||||
**After a power cut, an app can stay switched off — and the machine may look too early to notice.**
|
||||
It now knows the difference between an app the customer switched off and one that simply didn't come
|
||||
back: the customer's choice is written down, so an app meant to be running is started again however
|
||||
it went missing. What is left is timing — it looks about five seconds after starting, while the
|
||||
system is still coming up, and never looks again. *(R-157; the "decided the customer switched it off"
|
||||
half is fixed — R-166, 2026-08-02. The same guess still lives in one other place, for apps kept on a
|
||||
plug-in drive — R-170.)*
|
||||
## What's broken
|
||||
|
||||
**The off-site copy can be erased by the machine that made it** — the credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
@@ -69,6 +66,15 @@ It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)*
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-02** — Boot recovery finished. Both halves of the power-cut problem are closed: the
|
||||
machine records what the customer asked for, and it now waits for the system to finish starting
|
||||
before deciding what is missing. Six hard resets in a row, everything back every time. A hole the
|
||||
previous day's change had opened — the machine would have tried to start an app whose external
|
||||
drive was missing — was found by reading the code, reproduced on the demo box **before** anything
|
||||
was changed, and fixed the same day. No data ever reached the wrong disk, but only because of a
|
||||
filesystem accident nothing was relying on deliberately, which is why it was fixed rather than
|
||||
noted.
|
||||
|
||||
- **2026-08-02** — Fixed and then doubled: thirteen mechanical checks had built up across the four
|
||||
repositories and nothing ran most of them; two were failing quietly, one since 14 July. Both fixed.
|
||||
Every repository now has one command that runs all its checks, it runs automatically before every
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -120,10 +120,44 @@ Applied: a lost or corrupt marker means the app is not auto-restarted **by that
|
||||
the pre-v0.189.0 position, not a new hazard. A lost `app.yaml` already means the app is not deployed.
|
||||
**Nothing here may make an absent file more dangerous than a present one.**
|
||||
|
||||
### Not migrated
|
||||
### Boot recovery reads desired, and asks before it acts (v0.190.0)
|
||||
|
||||
`shouldRecreateOnBoot` (`internal/web/intermediary.go`) — the drive-backed boot gate — still derives
|
||||
desired from observed via `hasContainers`. Tracked as **R-170**.
|
||||
**Both boot gates now read desired state.** `bootrecon.isBootOrphan` (the R-52 sweep) and
|
||||
`shouldRecreateOnBoot` (the drive-backed recreate gate) answer the same question — *did the customer
|
||||
want this running?* — with the same three-way table, absent falling back to the pre-v0.190.0
|
||||
container count in both. Their agreement is pinned from both sides against one fixture table, because
|
||||
an import cycle prevents testing them together. R-170 closed the last gate that still guessed.
|
||||
|
||||
**The sweep observes a SETTLED fleet, not a single early sample.** It samples (name, state, container
|
||||
count) every 5 s, calls the fleet settled after 3 identical samples, and sweeps **once**, at the end.
|
||||
The window ends on settled or a 50 s budget, and the log says which. Two constraints bound it:
|
||||
|
||||
- `bootReconcileSettle + budget + one DefaultRetryDelay` must stay inside `deadAppBootGrace`, or a
|
||||
successful recovery stops being silent. This is arithmetic, pinned by a test.
|
||||
- **each sample must REFRESH first.** `GetStacks()` is the Manager's in-memory map, refreshed by the
|
||||
scheduler on its own 10 s cadence; sampling it faster without refreshing lets "settled" mean "the
|
||||
cache did not update". Found by live validation, not review.
|
||||
|
||||
A recovery completing after the grace emits a `LATE RECOVERY` warning naming the apps. The grace is
|
||||
**never** widened to make a late recovery look silent.
|
||||
|
||||
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
|
||||
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
|
||||
answer it, and the last two only became reachable once the window widened past T+5 s:
|
||||
|
||||
| Holder | Why starting would be wrong |
|
||||
|---|---|
|
||||
| the drive-absent gate | compose creates bind sources wherever the mountpoint points — the guest rootfs |
|
||||
| a quiesce | a running app inside a snapshot meant to be clean-shutdown-consistent |
|
||||
| an in-flight app-data operation | restarting an app under its own tar |
|
||||
|
||||
The rule is **not new** — the API's `startGatedByMissingDrive` already refused a customer's start on
|
||||
an absent drive. The sweep bypassed it by calling `Manager.StartStack` directly, which is what R-171
|
||||
closed. Held apps are reported separately from "still down": they are not a fault the sweep failed to
|
||||
fix, and reporting them as one is a false alarm.
|
||||
|
||||
**Read this before adding a fourth caller of `Manager.StartStack`.** That method has no gate of its
|
||||
own; every caller that is not the customer must decide for itself whether the app may run.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
# DIAG — does the R-166 boot sweep start an app whose data drive is absent?
|
||||
|
||||
**Date:** 2026-08-02 · **Box:** demo `felhom-pve` guest 9201 · **Controller:** v0.189.0
|
||||
**Question raised by:** reading the v0.189.0 diff, **not** by an incident. Diagnosed before any fix
|
||||
was written (task §9.2, "diagnose before theorising").
|
||||
|
||||
## Verdict
|
||||
|
||||
**CONFIRMED — the sweep starts it.** Observed directly, twice in the same run (both attempts).
|
||||
|
||||
**With a qualification that changes the severity but not the fix:** on this box the resulting
|
||||
`docker compose up -d` **failed** and **nothing was written** to the wrong disk. The mechanism that
|
||||
prevented the write is a **filesystem-permission accident that no code owns**, and the harm that DID
|
||||
occur is a different one: a false dead-app alarm for an app the drive gate is deliberately holding.
|
||||
|
||||
## The reasoning under test
|
||||
|
||||
`bootrecon` imports `stacks` alone and has no storage awareness. Since v0.189.0 `isBootOrphan`
|
||||
returns true for `desired_state: running` + zero containers. The drive-absent gate stops apps with
|
||||
`compose down` (leaving exactly zero containers) and never touches `desired_state`, because it is not
|
||||
the customer. `Manager.StartStack` has no drive gate. Therefore the sweep should start an app whose
|
||||
drive is absent.
|
||||
|
||||
## Method
|
||||
|
||||
The drive is a real USB disk (`/dev/sdb`), bound under the stable parent at
|
||||
`/mnt/felhom-drives/hdd_1`. The gate's absence signal is the agent's `BoundUnderParent`
|
||||
(`planDriveGates`, `intermediary.go:226`), so the drive was made absent **by unmounting it**, not by
|
||||
editing controller state — the state edit would have proven a different thing.
|
||||
|
||||
**Run 1 was contaminated and is reported because it produced a mechanism.** Unmounting only the
|
||||
parent bind was not enough: **the agent re-binds it within ~60 s** while `/dev/sdb` is still mounted
|
||||
at `/mnt/hdd_1`. In that run the drive gate's startup reconcile re-attached the drive and restarted
|
||||
the apps at `17:18:47`, **one second before** `bootrecon` looked at `17:18:48` — which then logged
|
||||
`no boot-orphaned apps`. That is a race that happened to go the safe way, **not** a disproof. Had it
|
||||
been reported as one, the conclusion would have been wrong.
|
||||
|
||||
Run 2 therefore unmounted **both** `/mnt/felhom-drives/hdd_1` and `/mnt/hdd_1`, and held them
|
||||
unmounted against the agent's healing for the duration.
|
||||
|
||||
## Preconditions, all verified before the observation
|
||||
|
||||
| # | Precondition | Observed |
|
||||
|---|---|---|
|
||||
| 1 | drive absent | `mount \| grep -c hdd_1` → **0** |
|
||||
| 2 | app has zero containers | `docker ps -a --filter name=calibre` → **0** |
|
||||
| 3 | intent still says running | `desired_state: running` |
|
||||
| 4 | app still deployed | `deployed: true` |
|
||||
|
||||
The gate stopped the apps itself first, on its own signal:
|
||||
|
||||
```
|
||||
17:22:07 [WARN] [gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 2 app(s): [calibre-web immich]
|
||||
```
|
||||
|
||||
`immich` was set to `desired_state: stopped` beforehand to scope the blast radius to one app; the
|
||||
gate is per-drive, so it stops every app on that drive regardless.
|
||||
|
||||
## The observation
|
||||
|
||||
```
|
||||
17:22:29 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded
|
||||
17:22:34 [INFO] [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s)
|
||||
17:22:34 [INFO] [stacks] Starting stack: calibre-web
|
||||
17:22:35 [WARN] [bootrecon] attempt 1/2: start "calibre-web" failed after 0.4s: exit code 1
|
||||
17:23:05 [WARN] [bootrecon] attempt 2/2: start "calibre-web" failed after 0.4s: exit code 1
|
||||
17:23:05 [WARN] [bootrecon] gave up after 2 attempt(s): recovered=[] still down=[calibre-web] (the dead-app alarm now owns these)
|
||||
```
|
||||
|
||||
**The sweep selected a drive-absent app and called `StartStack` on it.** That is the confirmation.
|
||||
|
||||
## What stopped the write, and why it must not be relied on
|
||||
|
||||
```
|
||||
Error response from daemon: error while creating mount source path
|
||||
'/mnt/felhom-drives/hdd_1/userdata/media/books':
|
||||
mkdir /mnt/felhom-drives/hdd_1/userdata: permission denied
|
||||
```
|
||||
|
||||
With the drive unbound, `/mnt/felhom-drives/hdd_1` is an empty directory on the host's `pve-root`,
|
||||
and it is **host-root-owned**:
|
||||
|
||||
```
|
||||
/mnt/felhom-drives uid=0 gid=0 mode=755
|
||||
/mnt/felhom-drives/hdd_1 uid=0 gid=0 mode=755
|
||||
```
|
||||
|
||||
Guest 9201 is an **unprivileged** LXC, so its container root is uid 100000 and cannot `mkdir` there.
|
||||
Confirmed no write occurred: `find /mnt/felhom-drives/hdd_1/` returned the directory alone, and
|
||||
`df -h /` was unchanged at `25G used / 28%` before and after.
|
||||
|
||||
**This protection is accidental.** Nothing in the controller chose it, no test pins it, and it rests
|
||||
on two conditions that are not guaranteed and are not checked anywhere:
|
||||
|
||||
1. the guest is unprivileged (a privileged guest maps root→0 and the `mkdir` succeeds);
|
||||
2. the stable-parent mountpoint directory is root-owned. When the drive **is** bound, that same path
|
||||
shows `uid=100000 gid=100000` — i.e. guest-writable. Any code path or agent version that
|
||||
pre-creates the mountpoint with guest ownership removes the protection silently.
|
||||
|
||||
It is one `chown` away from being gone, and its removal would be invisible until data landed on the
|
||||
wrong disk. It is therefore **not** a reason to leave the sweep unguarded.
|
||||
|
||||
## The harm that DID occur
|
||||
|
||||
Independent of the write question, and real on every box:
|
||||
|
||||
- the sweep burns **both** attempts and 30 s of retry delay on an app that cannot start by design;
|
||||
- it then hands the app to the **dead-app alarm** — `still down=[calibre-web] (the dead-app alarm now
|
||||
owns these)` — producing a **false alarm about an app the drive gate is deliberately holding**,
|
||||
which is exactly the noise class R-97/F-A1 exist to prevent;
|
||||
- it leaves a `Created` container behind on each attempt.
|
||||
|
||||
Before v0.189.0 none of this happened: `isBootOrphan` required `len(Containers) > 0`, and a
|
||||
gate-stopped app has zero. **This is a regression introduced by v0.189.0.**
|
||||
|
||||
## A second asymmetry found while restoring the box
|
||||
|
||||
The **API** start path already refuses this correctly. Restoring `immich` through the endpoint the UI
|
||||
calls, while the path was still flagged disconnected, returned:
|
||||
|
||||
```
|
||||
{"ok":false,"error":"A(z) /mnt/felhom-drives/hdd_1 tárhely jelenleg nem elérhető —
|
||||
az alkalmazás nem indítható, amíg a meghajtó vissza nem csatlakozik."}
|
||||
```
|
||||
|
||||
That is `startGatedByMissingDrive` (`internal/api/router.go`). So the controller already holds the
|
||||
rule "do not start an app whose drive is missing" — it is enforced on the customer's path and
|
||||
**bypassed by the sweep**, which calls `Manager.StartStack` directly. The fix is to give the sweep
|
||||
the same question to ask, not to invent a new rule.
|
||||
|
||||
## Consequence for the task
|
||||
|
||||
**Part 3 applies and is implemented first**, before the sweep's window is widened — a wider window
|
||||
makes both the false alarm and the (currently accident-blocked) write hazard wider. Fail-safe
|
||||
direction per §8.4: **cannot determine drive liveness → do not start.** Not starting is recoverable
|
||||
(the gate's `Return` branch restarts the app when the drive comes back, and the alarm reports it
|
||||
meanwhile); starting on an absent drive is not recoverable by anything automatic.
|
||||
|
||||
Register row: **R-171**, marked as a regression from v0.189.0.
|
||||
@@ -85,8 +85,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-138** | **A shared-zone `cf_api_token` is a zone-wide DNS-write capability on a customer's box** — written 0600 to `/opt/docker/stacks/traefik/.env` (`controller/internal/infra/infra.go:123`) | READY (S) | — | Today each box holds a token for a zone nobody else uses, so the blast radius is one customer. Under a shared customer zone, one compromised tester box could repoint every other tester's DNS. The ACME path is already switchable — an empty token selects HTTP-01 (`traefik.yml.tmpl`) — so the fix is policy plus a guard that refuses to hand a shared-zone customer a zone-scoped token. Same audit §5.2 | CC |
|
||||
| **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC |
|
||||
| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. **THE REFERRAL IS RESOLVED, 2026-08-02 — papra is deployed NOWHERE, so the template fix strands nothing and can be applied.** The referral existed because changing where the volume mounts moves live data: an installed papra writes `db.sqlite` into the container's writable layer, and a remount relocates the path out from under it. With no instance deployed there is no live data to move, so the cheaper leg — **the template mounts `/app/app-data`** — is takeable directly, without waiting on upstream to adopt `/app/data`. **Provenance, stated because it decides the row:** the observation is `docker ps -a` on demo-hp's **guest 9201** returning empty, supplied with the 2026-08-02 task; **this session did not re-measure** (documentation-only, every box fenced). **Scope of that evidence, honestly:** it covers guest 9201 — the guest papra was convicted on in Campaign 10 — and **no other customer's guest was enumerated**, so a re-check belongs in the task that edits the template, before it edits it. **Next action: apply the template fix in `app-catalog-felhom.eu` and re-run `scripts/catalog_gates.py`** (deliberately not done here — that repo was out of scope for this task). See R-161 (nothing runs the gate automatically) and R-159/R-160 | CC |
|
||||
| **R-157** | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms. MECHANISM B IS CLOSED; MECHANISM A REMAINS.** **(A) STILL OPEN:** the sweep runs ~5 s after controller start (`cmd/controller/main.go`, `runBootReconcile`) while docker is still restoring, sees "no boot-orphaned apps", and never re-checks — **intermittent, 3 of 6 hard resets**. R-166 did **not** touch the timing, so this is untouched and is now the whole of R-157. **(B) CLOSED by R-166 (controller v0.189.0, 2026-08-02):** an interruption mid-compose left the stack with **zero containers**, the signature bootrecon deliberately skipped as a user's Stop. The sweep now reads the customer's RECORDED intent (`desired_state` in `app.yaml`), so `running` + zero containers is an orphan and is recovered — **proven live on 9201**: containers removed out-of-band, controller restarted, `[bootrecon] 1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt. | **READY (S) — A only** | — | **A is now a pure timing fix and is the only thing left here:** a settle condition before the single observation, or one re-evaluation at the end of `deadAppBootGrace`, still bounded to the same attempts. Test the **consequence** (app running again before the grace expires), not the mechanism. Note that A and B were independent: B was a wrong signal, A is a premature observation, and B's fix makes A **more** visible (the sweep now has more it could recover, so observing too early costs more). Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`; B's closure in `felhom-controller/REPORT.md` (2026-08-02) | CC |
|
||||
| **R-170** | **The drive-backed boot gate still infers a customer's Stop from a container count — R-157 mechanism B, one gate over, for drive-backed apps only.** `shouldRecreateOnBoot` (`controller/internal/web/intermediary.go:131`) ends in `&& hasContainers`, sampled at `:425` as `len(st.Containers) > 0`. That term was added deliberately by **R-55** (controller v0.157.0) to stop the gate resurrecting an app the customer had stopped, and it was the right fix with the signal available then. R-166 replaced exactly that inference in `bootrecon` with the recorded `desired_state`, and **this gate was left on the old signal** — so for a drive-backed app, a power cut or interrupted deploy that leaves zero containers is still read as a deliberate stop by this path | **READY (S) — NEW 2026-08-02** | — | **NOT acted on by R-166, deliberately** — the task scoped `bootrecon` and named this gate nowhere, and changing a second boot path in the same release would have shipped two behaviour changes under one live validation. **Consequence is bounded, not zero:** `bootrecon` now recovers the same app a few seconds later on intent, so the observable defect is narrowed to the gate's own recreate-onto-the-drive step being skipped — which matters when the app needs recreating against a re-bound drive, not merely starting. **Fix shape:** replace `hasContainers` with `stacks.DesiredStateOf(st) != DesiredStateStopped` and keep the legacy fallback, i.e. the §8.1 table this gate already half-implements. **Established free:** `grep -ro "R-170\b" documentation/ *.md` → 0 hits (2026-08-02) | CC |
|
||||
| **R-157** | ~~**`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (B: controller v0.189.0; A: v0.190.0, 2026-08-02) | — | **Both mechanisms closed. (B)** the container-count signal → recorded intent (R-166). **(A)** the sweep looked ONCE at T+5 s, deriving candidates from a fleet docker was still restoring — 3 of 6 hard resets. Now a **settle-then-sweep window**: sample the fleet every 5 s, settled after 3 identical samples, sweep ONCE at the end; ends on settled OR a 50 s budget, and the log says which. **The budget is 50 s because a test rejected 60 s**: settle+budget+one 30 s retry must stay under the 90 s `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s. A window that genuinely overruns emits a `LATE RECOVERY` WARN naming the apps — the grace was NOT widened to hide it (§8.3). **A defect in the fix, found by live validation not review:** `GetStacks()` is the Manager's cache, refreshed by the scheduler every 10 s, so sampling it every 5 s without refreshing let "settled" mean "the cache did not update" — observed missing a container removed 5 s before the window closed. `sampleBootFleet` now refreshes first. **Live: 6/6 hard resets on the shipped build, every app back every time** (settle times 10/40/10/10/15/15 s — i.e. the window routinely waited 2–8× longer than the old fixed 5 s), plus a before/after on ONE app on ONE box: the pre-fix window logged `no boot-orphaned apps` for calibre-web at 18:08:35, the fixed one found and recovered it at 18:18:50 | — |
|
||||
| **R-170** | ~~**The drive-backed boot gate infers a customer's Stop from a container count.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | `shouldRecreateOnBoot` now reads `desired_state` with the SAME three-way table as `isBootOrphan`: `stopped` → never; `running` → recreate whatever the container count; **absent → exactly the pre-v0.190.0 `hasContainers` behaviour**. `presentStable` untouched and still load-bearing (an absent drive is never recreated here — the very term the boot sweep was missing, R-171). Its comment argued at length FOR the container count and was rewritten; a correct implementation under a comment arguing the opposite is worse than either alone. **The agreement is pinned from BOTH sides** against one fixture table (`TestBothBootGatesAgreeOnIntent` / `TestShouldRecreateOnBoot_AgreesWithBootrecon`) because the two gates cannot be called from one package without an import cycle. **Live on 9201, both halves in one reboot:** calibre-web (drive-backed, `running`, ZERO containers) → `recreating drive-backed app calibre-web`; immich (`stopped`) → `1 drive-backed app(s) left stopped on purpose` | — |
|
||||
| **R-171** | **The boot sweep started apps whose data drive was ABSENT — a regression introduced by v0.189.0, now FIXED.** Replacing `isBootOrphan`'s container-count term with recorded intent made a drive-gate-stopped app (`compose down` ⇒ zero containers, and the gate never touches `desired_state` because it is not the customer) read as a boot orphan | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | **Reasoned from the diff, then CONFIRMED on hardware before any fix was written** (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The sweep found and started calibre-web with its drive unmounted, burned both attempts and handed it to the dead-app alarm — **a false alarm about an app the drive gate is deliberately holding**. The *write* hazard did NOT materialise: compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is host-root-owned and the guest is unprivileged — **an accidental protection no code owns, no test pins, and one `chown` or one privileged guest away from gone**. Fix: new consumer-side seam `bootrecon.StartGate`, **fail-safe (cannot determine ⇒ do not start)**, wired in `main.go`; `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two cannot drift. **The rule is not new** — the API's `startGatedByMissingDrive` already refused this to the customer; the sweep bypassed it by calling `Manager.StartStack` directly. Widening the window (R-157 A) made two more holders reachable, so the same seam also refuses an app held by a **quiesce** or an **in-flight app-data operation** (§8.2), reusing `quiesce.SuppressedStacks()` and a new read-only `AppStopGuard.HeldStacks()`. Held apps report as `HeldByDrive`, never `StillDown` — that is the alarm's bucket. **ID established free:** `grep -ro "R-171\b" documentation/ *.md` → 0 hits before minting | — |
|
||||
| **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC |
|
||||
| **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `<project>_<name>` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC |
|
||||
| **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |
|
||||
|
||||
@@ -162,9 +162,10 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
|
||||
| R-154 | **`[first-boot]` is automated-install-only, and nothing in the tree said so** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | A PVE property, measured with a same-image control (`audits/SPIKE-universal-iso-3-2026-07-31.md` §2); recorded at `scripts/iso/pkg/build-deb.sh:6-11`. Superseded in practice by the `.deb` delivery route |
|
||||
| R-155 | **`iso-repack.sh` refused any ISO without `auto-installer-mode.toml`** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | **Narrowed, not deleted** — unchanged for `FELHOM_MENU=single` (`iso-repack.sh:121-128`), does not apply to `release` where the file's absence *is* gate G1. Do not remove it wholesale |
|
||||
| R-156 | **An app's data is neither persisted nor backed up, and it reports healthy** — a template mounts a volume at a path the app never writes, so data sits in the container's writable layer: lost on redeploy, tarred nightly as an empty dir, healthcheck green | S | **DETECTED + GATE SHIPPED; papra REFERRED** (catalog sweep, 2026-08-02) | Found by Campaign 10 on **papra**; the 53-template sweep convicted **gramps-web** and **wishlist** too and **fixed both**. The gate is `app-catalog-felhom.eu/scripts/check-volume-persistence.py` — a runtime probe (`docker diff` + mount occupancy + writability, canary self-test, fails closed); the detector and the gate are one program. **papra is NOT fixed**: the fix needs the app to use `/app/data` or the template to mount `/app/app-data`, so it is referred. Sweep evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` (53 probe.json). Ranking in `OPEN-ITEMS.md` |
|
||||
| R-157 | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — two mechanisms** | S | **B SHIPPED (controller v0.189.0, 2026-08-02) — A still READY** | R-52 built `internal/bootrecon` for exactly this (F5: two apps `Exited` ~18 h). **(B) CLOSED by R-166:** a mid-compose interruption left **zero containers**, the signature bootrecon skipped as a user Stop; the sweep now reads the recorded `desired_state` instead of counting containers, so `running` + zero containers is an orphan — **proven live on 9201** (containers removed out-of-band → `1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt). **(A) UNTOUCHED and now the whole item:** the sweep fires ~5 s after start while docker is still restoring and never re-checks — **3 of 6 hard resets**, intermittent, which is what a race predicts and a wrong predicate does not. R-166 deliberately did not change the timing. **B's fix makes A cost more**, since the sweep now has more it could legitimately recover. Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` |
|
||||
| R-157 | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover** | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.189.0 + v0.190.0, 2026-08-02)** | Both mechanisms closed. **B:** the container count → recorded intent (R-166). **A:** the single T+5 s observation → a settle-then-sweep window (sample every 5 s, settled after 3 identical samples, one sweep at the end, ends on settled OR a 50 s budget). Budget is 50 s because settle+budget+one retry must fit the 90 s `deadAppBootGrace`; a test rejected 60 s at 95 s. Late recoveries are REPORTED, not hidden by widening the grace. **The fix had its own defect, found live:** sampling the Manager's 10 s-refreshed cache let "settled" mean "the cache did not update" — `sampleBootFleet` now refreshes first. **6/6 hard resets clean on the shipped build**, plus a same-app before/after (missed 18:08:35, recovered 18:18:50) |
|
||||
| R-170 | **The drive-backed boot gate infers a Stop from a container count** | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | `shouldRecreateOnBoot` reads `desired_state` with the same three-way table as `isBootOrphan`; absent keeps the old `hasContainers` behaviour exactly; `presentStable` untouched. Agreement pinned from both sides against one fixture table. Live: calibre-web (`running`, zero containers) recreated and immich (`stopped`) left alone in the same reboot |
|
||||
| R-171 | **The boot sweep started apps whose data drive was ABSENT** — a regression from v0.189.0 | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | Reasoned from the diff, CONFIRMED on hardware first (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The write hazard was blocked only by an ACCIDENTAL filesystem permission (host-root-owned mountpoint + unprivileged guest); the false dead-app alarm was real on every box. New fail-safe `bootrecon.StartGate` seam, also covering quiesce and in-flight app-data operations (§8.2). The rule already existed on the API path (`startGatedByMissingDrive`); the sweep bypassed it |
|
||||
| R-166 | **App state gets a desired/observed model with its own store** (operator decision D-b) | M | **SHIPPED + PROVEN-LIVE — controller v0.189.0, 2026-08-02** | Tri-state `desired_state` in `app.yaml`, written ONLY by the customer's action; `bootrecon` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN** so legacy boxes keep byte-identical behaviour; running-only backfill; `backup.AppStopGuard` covers every stop→work→start window in its own marker file. **The durable fix for R-157 mechanism B.** Both blocking facts answered at source first: the crash-safe journal pattern DID already exist (quiesce, migrate) but covered **none** of the app-data path, and the SQLite store was rejected as a home because `metrics.db` is optional by design. `07`/`02` architecture docs carry the desired/in-flight/observed split. **Left for R-170:** the drive-backed boot gate still uses the container-count inference |
|
||||
| R-170 | **The drive-backed boot gate still infers a Stop from a container count** — R-157 mechanism B, one gate over | S | READY — NEW 2026-08-02 | `shouldRecreateOnBoot` (`internal/web/intermediary.go:131`) ends in `&& hasContainers`. Added deliberately by **R-55** and correct for the signal available then; R-166 replaced the same inference in `bootrecon` and left this path behind. Fix: read `desired_state` with the legacy fallback, i.e. the same §8.1 table. Bounded consequence — `bootrecon` recovers the app seconds later, so what is lost is the gate's recreate-against-the-rebound-drive step, not the start |
|
||||
| R-158 | **A local Tier-1 backup failure reaches no hub channel** — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, only the off-box leg calls it | S | READY — 2026-08-02 | Fifth *seam built but never wired*; **R-97's defect one tier over**. Only caller `cmd/controller/main.go:659`; the manager's seams are tier2/offbox/offbox-enlarge (`backup.go:33,37,58`), none for the unit capture. **Ranked below R-157 by measurement:** `/backups` DOES show `✗ Adatmentés sikertelen`, it persists across a second failure and clears on recovery, and `/backups/apps` shows the last good unit's true mtime — so it is a notification gap, not silent failure. **The specific fix: `/backups/apps` is the page you would check for one app, and the only one that never says.** Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md`, `audits/CAMPAIGN-10-closeout-2026-08-02.md` |
|
||||
| R-159 | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy** | XS | **SHIPPED** (`templates/wishlist/`, 2026-08-02) — filed for the CLASS | Image declares `VOLUME /usr/src/app/data`; template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns names only for compose-declared volumes, so `DumpAppVolumes` never sees an anonymous one. **The class is open:** any image `VOLUME` at an unmounted path is silent unbacked-up storage — **`immich-server` has one today** at `/data`, empty when measured. Proposed `REUSE.md` rule: a template mounts every path in `Config.Volumes`, or says why not |
|
||||
| R-160 | **gramps-web persisted three paths and wrote to none of them** | XS | **SHIPPED** (`templates/gramps-web/`, 2026-08-02) | `/app/data` appears nowhere in the image's env; the accounts DB and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths, the template three, one a phantom. **Severity above papra's:** papra loses documents the customer may hold elsewhere; gramps-web loses the artefact built inside the app, of which no other copy exists by construction |
|
||||
|
||||
Reference in New Issue
Block a user