NotifyStorageDisconnected and NotifyStorageReconnected were defined and called
from NOWHERE. Registered in allowedEventTypes, in DefaultEnabledEvents, and given
a Hungarian message on the hub -- and never invoked. A drive going absent produced
apps stopped, a WARN log and a UI badge, then silence on every channel. Verified
against the gitignored-cmd/ trap with a positive control. Fifth instance of this
class, found by E-2 Phase 0 rather than by a failure.
A drive that is ONLY a backup target has no apps to stop, so it was silent twice.
ReconcileDriveGates now calls both halves. When the absent drive is the whole-guest
backup target it raises the more specific backup_target_absent (error) instead --
never both, since two mails for one event trains people to ignore the channel --
and recovers as backup_target_restored (info, the existing pairing-gated pattern;
severityNotifies NOT widened). The recovery mirrors the alarm's choice or the
operator cannot match them.
Which drive is the target comes from the AGENT (/disks backup_target, >= 0.112.0),
not from our StoragePath.BackupTarget: that is customer INTENT, and on the two
boxes migrated by hand in E-1 the intent was never recorded while the drive really
is the target. An older agent omits the field -> false -> generic alarm, never a
wrong one.
Before this an absent backup target had NO prompt signal: the tier stays DUE
(targetStoragePresent checks name presence, never reachability), so the only
evidence was its own failure at the next due cycle, up to ~24h away. The R-100
shape.
Tests observe the WIRE, not a mock, because the failure class is "nothing
arrives": a real Notifier posts to an httptest hub and the test asserts the event
type and severity that actually went out.
MinAgent: 0.112.0
Green gate: build + vet + test rc=0 (27 packages), run separately from this commit.
gofmt -w across the controller tree (46 files) so gofmt -l is empty — disarms the
formatting landmine where a targeted edit + accidental gofmt -w swept ~46 unrelated
files. Pure formatting: whitespace + gofmt's optional-semicolon removal in reflowed
inline closures. One doc comment reworded ('' -> 'the empty string') to avoid gofmt's
Go-1.19 doc-comment typographic substitition ('' -> curly quote) muddying its meaning.
No build/vet/test behavior change.
shouldRecreateOnBoot keyed on Deployed+drive-present alone. Deployed stays
true across a Stop, so a drive-backed app the customer switched off was
silently restarted on every guest reboot (proven live: immich).
Requires len(Containers)>0 as well - R-52's existing-Exited vs absent
distinction. A UI Stop is compose down and removes the containers; a guest
that went down under a running app leaves them. Container STATE is still
deliberately NOT a filter: that would miss a not-yet-restarted or stuck-Exited
app, which is the bug the boot-id path exists to fix.
Evidence sampled before any recreate - recreate's own StopStack erases it.
Honoured Stops counted and logged separately from no-live-bind skips.
Controller-side of NAS network storage, proxying to agent A1 /netstorage/*. Distinct
'network' storage kind (no drive lifecycle); add/list/remove + per-share health UI;
unreachable NAS is a recoverable warning, never the drive missing/stop cascade; SMB
creds pass through to the agent, never persisted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
processGuestBootChange recreated the drive-backed app stacks but never re-synced
FileBrowser (base-infra, no HDD_PATH), so its drive mounts went stale after a
reboot. Now, AFTER pollLiveBinds confirms the live binds and the apps are
recreated, trigger go s.SyncFileBrowserMounts() so FileBrowser converges against
the now-live drives. Refactored into pure recreateDriveBackedApps(stacks, present,
recreate, syncFB). Tests: FB sync runs once after recreate (red-proofed companion);
runs even when nothing recreated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live diagnosis of drive-backed apps stuck Exited after a pct reboot pinned THREE
sub-causes, fixed together (hardening the existing processGuestBootChange, not a
parallel mechanism):
1. Agent-path blocker (live root cause): agentClient() returned "agent not
configured" (cfg.LocalAPI.Endpoint empty), so processGuestBootChange AND the
whole drive gate bailed at the first guard. bootstrap.json had a complete
local_api block, but MaybeIngest returned immediately on "already configured"
so a controller.yaml seeded before local_api existed never got the agent path.
Fix: MaybeIngest now calls ensureLocalAPI on the already-configured path,
merging local_api from bootstrap.json into the existing controller.yaml when
missing (no hub re-pull, config preserved; idempotent + fail-safe).
2. Boot-race readiness gate: processGuestBootChange sampled BoundUnderParent once
during fast startup, racing the ~18s rebind, recreated nothing, burned its
boot-id one-shot. Fix: gate on the REAL live in-guest bind -- driveBindLive
checks /mnt/felhom-drives/<drive> is a mountpoint in the controller's own /mnt
rslave /proc/self/mountinfo; pollLiveBinds waits for it (bounded ~120s) before
recreating via the normal pipeline. shouldRecreateOnBoot stays state-independent
so stuck-Exited create-time-failure apps are included.
3. Single-shot fragility: processGuestBootChange ran only once at startup; a
briefly-unreachable agent right after a guest reboot stranded recovery. Fix:
driveGateLoop runs it every periodic tick too (idempotent, boot-id gated).
Tests (non-hollow, pre-fix companions, red-proofed): pollLiveBinds waits then
reports live / never-live stays absent / single early sample misses; ensureLocalAPI
merges local_api into a configured controller.yaml that lacks it / no-ops when
present. Live-accepted with repeated pct reboot 9201.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sub-cause: on guest pct reboot, in-guest dockerd auto-starts unless-stopped
drive-backed apps ~18s BEFORE the agent re-binds the drive; the create-time
volume bind fails (mkdir /mnt/felhom-drives/<drive>/userdata: permission denied)
and RestartCount=0 means it's never retried -> stuck Exited. The existing
recovery (processGuestBootChange) RAN but raced the rebind: it sampled the
agent's BoundUnderParent once during fast startup (not live yet), recreated
nothing, and persisted the new boot-id -> burned its one-shot. The periodic gate
never recovered them either (first observation after the rebind -> no transition).
Fix (harden the existing mechanism, no parallel one): processGuestBootChange now
gates on the REAL live in-guest bind. driveBindLive checks whether
/mnt/felhom-drives/<drive> is an actual mountpoint in the controller's own /mnt
(rslave) /proc/self/mountinfo -- true only once the agent's bind propagated,
exactly when docker can recreate the app. pollLiveBinds waits for that (bounded
~120s, poll 2s; rebind lands ~18s) and only then recreates via the normal
pipeline, including stuck-Exited create-time-failure apps (shouldRecreateOnBoot
is state-independent). Single-flight; absent-after-window drives left to the gate;
host-reboot path unaffected; guest-only reboot path now covered.
Tests: pollLiveBinds waits through the rebind then reports live (recreate fires);
never-live drive stays absent; pre-fix companion (single early sample misses the
not-yet-live bind). Red-proofed against a no-wait single-sample.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
E1 caught it: filtering on State!=stopped missed apps docker hadn't auto-restarted
yet at the one-shot instant (5 apps exited after host reboot). Now recreates every
deployed present drive-backed app regardless of state (deployed=should run).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recreateBootStaleApps ran before the stack manager finished scanning (GetStacks
empty) so it found no apps; add a bounded wait for stacks before the one-time
boot-stale recreate. Deterministic guest-reboot convergence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recreateBootStaleApps recreates a present drive-backed app when boot-stale OR
exited/restarting/unhealthy (the recency-only gate missed already-exited apps).
Still skips healthy long-running + cleanly user-stopped apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes guest-reboot convergence. driveGateLoop runs recreateBootStaleApps once
at startup: deployed drive-backed apps whose drive is present (BoundUnderParent)
and whose containers started recently (fresh guest boot, not a controller-only
restart) are recreated (down+up) onto the re-propagated drive. Paired with agent
v0.35.0's drive re-propagation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The drive-absent gate treats a stable path usable only when bound under the parent
(BoundUnderParent), not merely host-mounted. Makes a host reboot converge: apps
stay gated until the agent binds the drive under the parent, then are restarted
(recreated) on the populated path. Test updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix caught live: planDriveGates falsely marked the internal SSD path
/mnt/sys_drive/felhom-data disconnected (agent never reports it), which would
block starting SSD-resident apps. Gate now skips non-/mnt/felhom-drives/ paths.
Regression case added. No apps were stopped (none depended on the SSD path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drives are visible in-guest only at the STABLE /mnt/felhom-drives/<name>; the
registered path + HDD_PATH + FileBrowser source repoint there while agent calls
map back to raw /mnt/<name> (agentWhere). Enroll binds-under-parent before
register. Drive-absent GATE (planDriveGates + 30s driveGateLoop) stops/blocks
apps when a drive vanishes and auto-restarts on return; start-gate refuses start
when the drive is absent. H1 endpoints (disconnect/reconnect/restart-apps) routed
onto host-side ops. Non-hollow tests + companions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin