R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix
was written). Replacing isBootOrphan's container-count term with recorded intent
made a drive-gate-stopped app read as a boot orphan: the gate stops apps with
`compose down` (zero containers) and never touches desired_state, because it is
not the customer. Observed on 9201 with the drive held unmounted — the sweep
found and started it, burned both attempts, and handed it to the dead-app alarm.
The write hazard did not materialise (the unbound mountpoint is host-root-owned
and the guest is unprivileged) but that protection is accidental and untested.
New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not
start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive
already refuses this; the sweep bypassed it.
R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a
fleet docker was still restoring — three of six hard resets. Now a settle-then-
sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at
the end; ends on settled or a 50s budget, and the log says which. The budget is
50s because settle+budget+one retry must stay under the 90s dead-app grace — a
test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn
rather than the grace being widened to hide it.
Widening the window made two more holders reachable, so the one gate covers all
three: an absent drive, a quiesce, and an in-flight app-data operation — reusing
quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks().
R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way
table; absent keeps the old hasContainers behaviour exactly. Its comment argued
for the container count and was rewritten. presentStable is untouched. The two
gates' agreement is pinned from both sides against one fixture table.
27/27 packages green; 6 red-proofs observed FAIL then restored.
Caught before deploy by tracing a.Path back to its source, not by a failure. The
0.184.0 image is superseded and must not be shipped.
ReconcileDriveGates resolves the target as isTarget[a.Path], and a.Path is the
REGISTERED StoragePath -- for an external drive that is the GUEST path
/mnt/felhom-drives/<name>, not the agent's host MountPath (/mnt/<name>) that
/disks reports. driveTargetByPath keyed on MountPath alone, so the lookup never
matched: every absent drive, the target included, fell through to the generic
storage_disconnected.
The alarm would have looked wired, passed its own unit tests, shipped, and been
silently wrong on exactly the drive it exists for -- the same defect class E-2b
was opened to fix, one level down.
Now keyed under BOTH paths, mirroring planDriveGates which already registers
present[] under GuestPath and MountPath for the same reason.
Red-proof: MountPath-only keying fails with "the backup target is not resolvable
by its GUEST path -- the gate passes a.Path (the registered StoragePath), so the
backup-target branch would never fire".
Green gate: build + vet + test rc=0, run separately from this commit.
NotifyStorageDisconnected and NotifyStorageReconnected were defined and called
from NOWHERE. Registered in allowedEventTypes, in DefaultEnabledEvents, and given
a Hungarian message on the hub -- and never invoked. A drive going absent produced
apps stopped, a WARN log and a UI badge, then silence on every channel. Verified
against the gitignored-cmd/ trap with a positive control. Fifth instance of this
class, found by E-2 Phase 0 rather than by a failure.
A drive that is ONLY a backup target has no apps to stop, so it was silent twice.
ReconcileDriveGates now calls both halves. When the absent drive is the whole-guest
backup target it raises the more specific backup_target_absent (error) instead --
never both, since two mails for one event trains people to ignore the channel --
and recovers as backup_target_restored (info, the existing pairing-gated pattern;
severityNotifies NOT widened). The recovery mirrors the alarm's choice or the
operator cannot match them.
Which drive is the target comes from the AGENT (/disks backup_target, >= 0.112.0),
not from our StoragePath.BackupTarget: that is customer INTENT, and on the two
boxes migrated by hand in E-1 the intent was never recorded while the drive really
is the target. An older agent omits the field -> false -> generic alarm, never a
wrong one.
Before this an absent backup target had NO prompt signal: the tier stays DUE
(targetStoragePresent checks name presence, never reachability), so the only
evidence was its own failure at the next due cycle, up to ~24h away. The R-100
shape.
Tests observe the WIRE, not a mock, because the failure class is "nothing
arrives": a real Notifier posts to an httptest hub and the test asserts the event
type and severity that actually went out.
MinAgent: 0.112.0
Green gate: build + vet + test rc=0 (27 packages), run separately from this commit.
gofmt -w across the controller tree (46 files) so gofmt -l is empty — disarms the
formatting landmine where a targeted edit + accidental gofmt -w swept ~46 unrelated
files. Pure formatting: whitespace + gofmt's optional-semicolon removal in reflowed
inline closures. One doc comment reworded ('' -> 'the empty string') to avoid gofmt's
Go-1.19 doc-comment typographic substitition ('' -> curly quote) muddying its meaning.
No build/vet/test behavior change.
shouldRecreateOnBoot keyed on Deployed+drive-present alone. Deployed stays
true across a Stop, so a drive-backed app the customer switched off was
silently restarted on every guest reboot (proven live: immich).
Requires len(Containers)>0 as well - R-52's existing-Exited vs absent
distinction. A UI Stop is compose down and removes the containers; a guest
that went down under a running app leaves them. Container STATE is still
deliberately NOT a filter: that would miss a not-yet-restarted or stuck-Exited
app, which is the bug the boot-id path exists to fix.
Evidence sampled before any recreate - recreate's own StopStack erases it.
Honoured Stops counted and logged separately from no-live-bind skips.
Controller-side of NAS network storage, proxying to agent A1 /netstorage/*. Distinct
'network' storage kind (no drive lifecycle); add/list/remove + per-share health UI;
unreachable NAS is a recoverable warning, never the drive missing/stop cascade; SMB
creds pass through to the agent, never persisted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
processGuestBootChange recreated the drive-backed app stacks but never re-synced
FileBrowser (base-infra, no HDD_PATH), so its drive mounts went stale after a
reboot. Now, AFTER pollLiveBinds confirms the live binds and the apps are
recreated, trigger go s.SyncFileBrowserMounts() so FileBrowser converges against
the now-live drives. Refactored into pure recreateDriveBackedApps(stacks, present,
recreate, syncFB). Tests: FB sync runs once after recreate (red-proofed companion);
runs even when nothing recreated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live diagnosis of drive-backed apps stuck Exited after a pct reboot pinned THREE
sub-causes, fixed together (hardening the existing processGuestBootChange, not a
parallel mechanism):
1. Agent-path blocker (live root cause): agentClient() returned "agent not
configured" (cfg.LocalAPI.Endpoint empty), so processGuestBootChange AND the
whole drive gate bailed at the first guard. bootstrap.json had a complete
local_api block, but MaybeIngest returned immediately on "already configured"
so a controller.yaml seeded before local_api existed never got the agent path.
Fix: MaybeIngest now calls ensureLocalAPI on the already-configured path,
merging local_api from bootstrap.json into the existing controller.yaml when
missing (no hub re-pull, config preserved; idempotent + fail-safe).
2. Boot-race readiness gate: processGuestBootChange sampled BoundUnderParent once
during fast startup, racing the ~18s rebind, recreated nothing, burned its
boot-id one-shot. Fix: gate on the REAL live in-guest bind -- driveBindLive
checks /mnt/felhom-drives/<drive> is a mountpoint in the controller's own /mnt
rslave /proc/self/mountinfo; pollLiveBinds waits for it (bounded ~120s) before
recreating via the normal pipeline. shouldRecreateOnBoot stays state-independent
so stuck-Exited create-time-failure apps are included.
3. Single-shot fragility: processGuestBootChange ran only once at startup; a
briefly-unreachable agent right after a guest reboot stranded recovery. Fix:
driveGateLoop runs it every periodic tick too (idempotent, boot-id gated).
Tests (non-hollow, pre-fix companions, red-proofed): pollLiveBinds waits then
reports live / never-live stays absent / single early sample misses; ensureLocalAPI
merges local_api into a configured controller.yaml that lacks it / no-ops when
present. Live-accepted with repeated pct reboot 9201.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sub-cause: on guest pct reboot, in-guest dockerd auto-starts unless-stopped
drive-backed apps ~18s BEFORE the agent re-binds the drive; the create-time
volume bind fails (mkdir /mnt/felhom-drives/<drive>/userdata: permission denied)
and RestartCount=0 means it's never retried -> stuck Exited. The existing
recovery (processGuestBootChange) RAN but raced the rebind: it sampled the
agent's BoundUnderParent once during fast startup (not live yet), recreated
nothing, and persisted the new boot-id -> burned its one-shot. The periodic gate
never recovered them either (first observation after the rebind -> no transition).
Fix (harden the existing mechanism, no parallel one): processGuestBootChange now
gates on the REAL live in-guest bind. driveBindLive checks whether
/mnt/felhom-drives/<drive> is an actual mountpoint in the controller's own /mnt
(rslave) /proc/self/mountinfo -- true only once the agent's bind propagated,
exactly when docker can recreate the app. pollLiveBinds waits for that (bounded
~120s, poll 2s; rebind lands ~18s) and only then recreates via the normal
pipeline, including stuck-Exited create-time-failure apps (shouldRecreateOnBoot
is state-independent). Single-flight; absent-after-window drives left to the gate;
host-reboot path unaffected; guest-only reboot path now covered.
Tests: pollLiveBinds waits through the rebind then reports live (recreate fires);
never-live drive stays absent; pre-fix companion (single early sample misses the
not-yet-live bind). Red-proofed against a no-wait single-sample.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
E1 caught it: filtering on State!=stopped missed apps docker hadn't auto-restarted
yet at the one-shot instant (5 apps exited after host reboot). Now recreates every
deployed present drive-backed app regardless of state (deployed=should run).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recreateBootStaleApps ran before the stack manager finished scanning (GetStacks
empty) so it found no apps; add a bounded wait for stacks before the one-time
boot-stale recreate. Deterministic guest-reboot convergence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recreateBootStaleApps recreates a present drive-backed app when boot-stale OR
exited/restarting/unhealthy (the recency-only gate missed already-exited apps).
Still skips healthy long-running + cleanly user-stopped apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes guest-reboot convergence. driveGateLoop runs recreateBootStaleApps once
at startup: deployed drive-backed apps whose drive is present (BoundUnderParent)
and whose containers started recently (fresh guest boot, not a controller-only
restart) are recreated (down+up) onto the re-propagated drive. Paired with agent
v0.35.0's drive re-propagation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The drive-absent gate treats a stable path usable only when bound under the parent
(BoundUnderParent), not merely host-mounted. Makes a host reboot converge: apps
stay gated until the agent binds the drive under the parent, then are restarted
(recreated) on the populated path. Test updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix caught live: planDriveGates falsely marked the internal SSD path
/mnt/sys_drive/felhom-data disconnected (agent never reports it), which would
block starting SSD-resident apps. Gate now skips non-/mnt/felhom-drives/ paths.
Regression case added. No apps were stopped (none depended on the SSD path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drives are visible in-guest only at the STABLE /mnt/felhom-drives/<name>; the
registered path + HDD_PATH + FileBrowser source repoint there while agent calls
map back to raw /mnt/<name> (agentWhere). Enroll binds-under-parent before
register. Drive-absent GATE (planDriveGates + 30s driveGateLoop) stops/blocks
apps when a drive vanishes and auto-restarts on return; start-gate refuses start
when the drive is absent. H1 endpoints (disconnect/reconnect/restart-apps) routed
onto host-side ops. Non-hollow tests + companions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin