felhom.eu/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md

DIAG — does the R-166 boot sweep start an app whose data drive is absent?

Date: 2026-08-02 · Box: demo felhom-pve guest 9201 · Controller: v0.189.0 Question raised by: reading the v0.189.0 diff, not by an incident. Diagnosed before any fix was written (task §9.2, "diagnose before theorising").

Verdict

CONFIRMED — the sweep starts it. Observed directly, twice in the same run (both attempts).

With a qualification that changes the severity but not the fix: on this box the resulting docker compose up -d failed and nothing was written to the wrong disk. The mechanism that prevented the write is a filesystem-permission accident that no code owns, and the harm that DID occur is a different one: a false dead-app alarm for an app the drive gate is deliberately holding.

The reasoning under test

bootrecon imports stacks alone and has no storage awareness. Since v0.189.0 isBootOrphan returns true for desired_state: running + zero containers. The drive-absent gate stops apps with compose down (leaving exactly zero containers) and never touches desired_state, because it is not the customer. Manager.StartStack has no drive gate. Therefore the sweep should start an app whose drive is absent.

Method

The drive is a real USB disk (/dev/sdb), bound under the stable parent at /mnt/felhom-drives/hdd_1. The gate's absence signal is the agent's BoundUnderParent (planDriveGates, intermediary.go:226), so the drive was made absent by unmounting it, not by editing controller state — the state edit would have proven a different thing.

Run 1 was contaminated and is reported because it produced a mechanism. Unmounting only the parent bind was not enough: the agent re-binds it within ~60 s while /dev/sdb is still mounted at /mnt/hdd_1. In that run the drive gate's startup reconcile re-attached the drive and restarted the apps at 17:18:47, one second before bootrecon looked at 17:18:48 — which then logged no boot-orphaned apps. That is a race that happened to go the safe way, not a disproof. Had it been reported as one, the conclusion would have been wrong.

Run 2 therefore unmounted both /mnt/felhom-drives/hdd_1 and /mnt/hdd_1, and held them unmounted against the agent's healing for the duration.

Preconditions, all verified before the observation

#	Precondition	Observed
1	drive absent	mount | grep -c hdd_1 → 0
2	app has zero containers	docker ps -a --filter name=calibre → 0
3	intent still says running	desired_state: running
4	app still deployed	deployed: true

The gate stopped the apps itself first, on its own signal:

17:22:07 [WARN] [gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 2 app(s): [calibre-web immich]

immich was set to desired_state: stopped beforehand to scope the blast radius to one app; the gate is per-drive, so it stops every app on that drive regardless.

The observation

17:22:29 [INFO] [stacks]     desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded
17:22:34 [INFO] [bootrecon]  Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s)
17:22:34 [INFO] [stacks]     Starting stack: calibre-web
17:22:35 [WARN] [bootrecon]  attempt 1/2: start "calibre-web" failed after 0.4s: exit code 1
17:23:05 [WARN] [bootrecon]  attempt 2/2: start "calibre-web" failed after 0.4s: exit code 1
17:23:05 [WARN] [bootrecon]  gave up after 2 attempt(s): recovered=[] still down=[calibre-web] (the dead-app alarm now owns these)

The sweep selected a drive-absent app and called StartStack on it. That is the confirmation.

What stopped the write, and why it must not be relied on

Error response from daemon: error while creating mount source path
  '/mnt/felhom-drives/hdd_1/userdata/media/books':
  mkdir /mnt/felhom-drives/hdd_1/userdata: permission denied

With the drive unbound, /mnt/felhom-drives/hdd_1 is an empty directory on the host's pve-root, and it is host-root-owned:

/mnt/felhom-drives        uid=0 gid=0 mode=755
/mnt/felhom-drives/hdd_1  uid=0 gid=0 mode=755

Guest 9201 is an unprivileged LXC, so its container root is uid 100000 and cannot mkdir there. Confirmed no write occurred: find /mnt/felhom-drives/hdd_1/ returned the directory alone, and df -h / was unchanged at 25G used / 28% before and after.

This protection is accidental. Nothing in the controller chose it, no test pins it, and it rests on two conditions that are not guaranteed and are not checked anywhere:

1. the guest is unprivileged (a privileged guest maps root→0 and the mkdir succeeds);
2. the stable-parent mountpoint directory is root-owned. When the drive is bound, that same path shows uid=100000 gid=100000 — i.e. guest-writable. Any code path or agent version that pre-creates the mountpoint with guest ownership removes the protection silently.

It is one chown away from being gone, and its removal would be invisible until data landed on the wrong disk. It is therefore not a reason to leave the sweep unguarded.

The harm that DID occur

Independent of the write question, and real on every box:

• the sweep burns both attempts and 30 s of retry delay on an app that cannot start by design;
• it then hands the app to the dead-app alarm — still down=[calibre-web] (the dead-app alarm now owns these) — producing a false alarm about an app the drive gate is deliberately holding, which is exactly the noise class R-97/F-A1 exist to prevent;
• it leaves a Created container behind on each attempt.

Before v0.189.0 none of this happened: isBootOrphan required len(Containers) > 0, and a gate-stopped app has zero. This is a regression introduced by v0.189.0.

A second asymmetry found while restoring the box

The API start path already refuses this correctly. Restoring immich through the endpoint the UI calls, while the path was still flagged disconnected, returned:

{"ok":false,"error":"A(z) /mnt/felhom-drives/hdd_1 tárhely jelenleg nem elérhető —
 az alkalmazás nem indítható, amíg a meghajtó vissza nem csatlakozik."}

That is startGatedByMissingDrive (internal/api/router.go). So the controller already holds the rule "do not start an app whose drive is missing" — it is enforced on the customer's path and bypassed by the sweep, which calls Manager.StartStack directly. The fix is to give the sweep the same question to ask, not to invent a new rule.

Consequence for the task

Part 3 applies and is implemented first, before the sweep's window is widened — a wider window makes both the false alarm and the (currently accident-blocked) write hazard wider. Fail-safe direction per §8.4: cannot determine drive liveness → do not start. Not starting is recoverable (the gate's Return branch restarts the app when the drive comes back, and the alarm reports it meanwhile); starting on an absent drive is not recoverable by anything automatic.

Register row: R-171, marked as a regression from v0.189.0.