deadapp-check had no observable at default info level: its per-cycle line goes
through Scheduler.dbg(), gated on logging.level==debug, so on a default box it is
never PRODUCED (not merely filtered) and cannot reach the always-DEBUG ring. A
30s interval also puts it on the scheduler's quiet path. 'No alarms' was
therefore indistinguishable from 'the detector never ran' — which undermines
confidence in the F-CRIT-1 fix in the field.
A periodic summary, not a line per run: at 30s a per-run line is 2880 lines/day,
which is why the original author chose silence. Every 20th scan (~10 min) emits
one INFO with the scan count, apps evaluated and apps down. A test pins the
cadence so it cannot be widened into uselessness.
Also corrects the 'unquiesce guaranteed by defer' comment — fault 10 established
the guarantee is the crash marker plus Recover().
Six red-proofs with observed failure text, the bounds chosen for contention
(15m retry / 3h alarm) justified against the agent's own 120m PBS restore-test
ceiling and the 12m01s longest observed restore-test, and the live replay of
both findings on demo-hp with the hub DB as arbiter:
- failed restart alarmed 9s after grace expiry; banner shows (stopped)
- a deliberate user stop stayed silent through 9 dead-app scans
- 409 contention produced 0 operator emails (8 -> 8)
- a real transport failure produced 1 (8 -> 9)
Same box, same event type; the only difference is 409 vs a genuine error.
F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
1. restartAll returned nothing, so the failure was logged and dropped and no
caller could learn a customer's app had not come back. It now returns the
stacks that failed; both call sites record the outcome.
2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
means the user stopped it'). The quiesce loop stops stacks by the same
compose-down path, so a failed restart is also StateStopped and was
whitelisted into silence. Loop.FailedRestarts() is now the only thing that
lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).
F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.
Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).
Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
A witness in a _test.go file fires on go test and go vet but NOT on go build
alone — and a build-only step is exactly how the R-88 Part 2 near-miss would have
shipped. Moved beside the type it pins, and added one for AgentVersionReporter.
No defect found: quiesceBackend and *Client both satisfy their interfaces today.
No version bump, no deploy — compile-time only.
Records the valve-licence table, why a missing field means legacy rather than
unknown, and the near-miss: TieredBackend is runtime-asserted, so the signature
change built and vetted clean while the adapter silently stopped satisfying it.
MinAgent: 0.105.0. scheduledRunAllowed fired on any nil age; it now requires a
licence from valveLicensed, which grants it for AgeStateAbsent and for a LEGACY
agent, and refuses it for AgeStateUnknown. An unreadable storage no longer
masquerades as a first-ever backup and no longer quiesces apps outside the window.
A missing wire field means legacy, not unknown — deliberately. Treating it as
unknown would stop the valve firing on un-upgraded boxes and starve genuinely new
ones. Degrade logged once; unrecognised future values also map to legacy.
Caught in passing: TieredBackend is satisfied by a RUNTIME assertion, so the
signature change compiled and vetted clean while quiesceBackend silently stopped
satisfying it — which would have degraded every box to the single-tier path with
no error. Added a compile-time witness.
Also corrects the notifier comment that claimed operator-only came from a missing
customerMessages entry; enforcement is hub-side operatorOnlyEvents (hub 0.79.0).
Includes the cooldown keying finding (per-event-type only; fixed narrowly with a
tier suffix), the 180s grace window derivation, and the end-to-end event evidence
with its control: the new type 200s, a bogus type 400s, and notification_log shows
one operator row and zero customer rows.
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest
backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import),
wired by an init-only setter because main.go builds the notifier after the loop.
Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and
recovery rides recordSuccess's existing bool. Uses NEW operator-only event types;
reusing backup_failed would have emailed the customer in Hungarian about a backup
they cannot act on, since it has a customerMessages entry and is in live
enabled_events. Requires hub >= v0.78.0.
R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how
BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE,
consumed at the same single derivation point. 180s grace, derived from the deploy
flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app
that genuinely fails to come back still alarms.
The quiesce loop is silent by design when nothing is due, so 'no error lines' is
not evidence. The observable that is: every /backup/due makes the agent read that
tier's storage content, which lands in pveproxy/access.log. Both tiers polled
every 5 min, both HTTP 200, tick phase consistent with the deploy restart.
Includes the Phase 0 findings: the nil is agent-side (case a, a type boundary
that cannot represent 'unknown'), the four restarts were deliberate rather than
self-update, and the loop produced zero backup_failed events because quiesce
never imports notify (filed R-97).
internal/quiesce had no failure counter, no backoff and no breaker, and the driver
is a plain 5-minute ticker — so a tier that was due and kept failing stopped and
restarted every customer app stack every 5 minutes indefinitely. Live on
demo-felhom 2026-07-27: three cycles in eleven minutes against an unreachable PBS
tier; it ended only because PBS recovered.
The breaker gates the QUIESCE, not the backup — the harm was the outage taken to
attempt it, so backed-off tiers are dropped before any stack is stopped. Per
target (a broken offsite tier must not suppress a healthy local one), 15m→30m→
1h→2h→4h capped, reset on success, never permanent, never applied to TriggerNow,
and stillRunning is not a failure. State is in-memory on purpose: forgetting a
backoff costs one attempt; persisting one could outlive the fix.
Part 3 invariant recorded on scheduledRunAllowed — a missing value means UNKNOWN,
and only a positive 'never' may fire the safety valve. Fourth instance of the
class (hub v0.12.0, v0.73.0, R-81, R-88).
Part 2 (unknown != never) is NOT in this commit: the agent returns byte-identical
responses for 'read errored' and 'never backed up', so the controller cannot tell
them apart. That needs an agent wire change and is tracked separately.
The section read 'NOT deployed'. True when written (17:56 CEST), stale 26 min
later when I deployed (18:22 CEST) and did not update it. REPORT.md is defined
as 'overwrite with the most recent state', so an artifact this project validates
against was left asserting the opposite of reality.
Precision matters on the apparent contradiction: a 17:01 quiesce log 'before' a
17:56 commit is a TIMEZONE artifact (controller logs UTC, git commits CEST), not
evidence of a false claim. In one clock: REPORT 17:56 -> deploy 18:22 -> quiesce
19:01. Consistent. That host=CEST/controller=UTC mismatch is a recorded project
trap and caught me once earlier the same day.
Also records the live multi-tier quiesce evidence and what remains unproven.
The multi-tier path only engages against an agent serving /backup/tiers, and the
agent rollout was still mid-flight, so both production controllers remain on
v0.173.0. Names the three things that are therefore unproven live.
Operator ruling 2026-07-26: let the first backup run as long as needed; other
backups shouldn't start until finished.
A first FULL offsite snapshot runs for hours, far past max_quiesce. When that
bound elapses the app resumes (unchanged), but the loop then started the NEXT
tier while the first was still uploading. Now it breaks and defers the rest to
a later poll — vzdump still holds the guest lock, so the second start would be
refused by the agent (409, v0.99.0) or fail on the lock, and a failed backup
never satisfies a cadence, so the tier would retry into the same wall forever.
pollTier returns (phase, stillRunning, err). The app still resumes exactly once.
Red-proof observed and restored; full suite green.
MinAgent UNCHANGED — degrades gracefully against ANY older agent.
The agent gained per-target tiers in v0.97.0. The controller owns quiescing,
so the multi-tier schedule is reconciled here: every due tier is collected up
front and run inside ONE quiesce window (one stop, N sequential backups, one
resume). Two cycles on the weekly night would mean two app outages for one
night's work.
Dedup rule: local-only -> one quiesce; PBS-only -> one quiesce; BOTH due ->
ONE window with both backups inside; neither -> no quiesce.
- quiesce.TieredBackend + BackupTier + ErrTiersUnsupported (optional extension)
- agentapi: BackupTiers/BackupDueFor/StartBackupFor/BackupStatusFor;
targetQuery("") yields an EMPTY suffix so untargeted hits the pre-R-82 route
byte-for-byte
- Loop.resolveDueTiers = the dedup rule in one place, agent order preserved
- quiesceAndPollTiers + pollTier: app stays quiesced until the LAST tier
snapshots (resuming earlier loses app-consistency on the DR tier). Consequence
stated in the docs: both-due-night downtime = first tier's full backup + last
tier's snapshot, which is why tiers run fast-first.
- Manual 'Mentes most' covers EVERY tier, due-ness ignored.
- Window-gate safety valve now uses the OLDEST due tier, so a stale DR tier
cannot be starved by a fresher local one.
Capability detection: /backup/tiers 404 = pre-R-82 agent (the documented
route-probe mechanism). Not a featureProbes row on purpose — the loop needs the
tier LIST, not a yes/no. Degrade logged exactly once per process.
Tests +11, full suite green. Red-proofs #2 and #3 observed and restored.
Source: felhom.eu/documentation/audits/DIAG-agent-channel-2026-07-26.md
bootstrap.DetectEndpointDrift names a controller.yaml vs bootstrap.json
local_api.endpoint divergence -- one ERROR carrying BOTH values and BOTH paths,
its own event type local_api_endpoint_drift, and its own Hungarian banner shown
ABOVE the channel banner because drift is the cause and "agent unreachable" the
symptom. It writes NOTHING: reconciling from bootstrap.json would clobber a
correct controller.yaml on any half-provisioned or hand-repaired guest, so the
authority ruling is deferred to R-78. Fail-safe silent on absent/unparseable/
incomplete bootstrap and on an empty endpoint (ensureLocalAPI's fill-if-missing
path is untouched). Fingerprint compared as a BOOLEAN only; token never
compared, logged or exposed.
EffectiveProtected now gates samba on Enabled && UserSet, mirroring BOTH of
reconcileSambaAt's early returns, and the doc comment is corrected in the same
change -- it claimed "detection and deployment agree in both directions" while
citing only !smb.Enabled, an assertion that went false when !smb.UserSet was
added. Not over-suppressed: sharing on WITH a password and a dead container
still alarms.
Channel log: the debounce placeholder is stateUnconfirmed (rendered "unseeded")
instead of "up", so a born-down channel no longer logs "up->down" and orUnseeded
stops being dead code. Logging only -- the placeholder is still matched in the
re-arm condition, so F2 born-down alerting is byte-for-byte unchanged and all
nine pre-existing channelhealth tests pass.
Tests 951 -> 959, all green. Red-proofs A (both directions), E and F.
MinAgent unchanged; felhom-agent untouched.
Found on the demo-hp live leg: with import, import/paperless and import/calibre
in the carry-list, the derived skeleton RE-CREATES a per-drive drop-zone on every
drive forever — the dead lookalike the canonical root exists to remove, and one
that is never backed up (class: excluded).
Not a zero-removals violation: nothing deletes what an existing box has. Both
demo boxes' old drop-zones were verified to hold zero files before the change.
Found on the demo-felhom deploy leg: ensuring only <sysNS>/userdata/import left
its parent at 755 root:root, because EnsureUserdataDir MkdirAll's intermediates
at plain 0755 and chmods only the leaf. That made the system drive's userdata
root the one on the box outside the 2775/gid-1000 convention.
${IMPORT_PATH} = <system namespace root>/userdata/import — ONE drop-zone per box,
on the system drive, injected at BOTH compose-env builders with NO per-drive
fallback (unresolvable leaves it unset so compose fails loudly rather than
quietly building a second, dead drop-zone).
Third BindRoot (RootImport) + Import list in BackupSpec, extended through
ValidateBackupSpec/ClassifyBinds. Load-bearing: a stale `userdata: import/<app>`
entry against the moved bind would be a WHOLE-BLOCK reject, taking the app's
mandatory hdd classification with it.
Exhaustive-root audit: resolveAbs/structuralGuard/ComputeCaptureSet/
ComputeFabBuckets now take importRoot explicitly (an import bind resolved
against hddPath would name a directory on the wrong drive); unresolvable is
refused loudly into Skipped. GetImportRoot added to both provider interfaces.
Catalog-derived skeleton: UserdataSkeleton() -> UserdataSkeletonCarry() +
BuildUserdataSkeleton(), SORTED. The carry-list makes zero-removals true by
construction (`documents` is in no catalog app but on both boxes) and is the
fresh-box floor. The sort is not tidiness: the naive map-order derivation
measured 20 distinct outputs from 20 identical runs, which with fbNeedsRecreate
is a fleet-wide FileBrowser restart loop.
One authoritative compose parser: ParseComposeUserdataMounts now delegates to
ParseComposeClassifiableBinds. Import root excluded from per-app migration.
Surfaces: FileBrowser /srv/beolvasas source; app-page "Hova tegyem a fajlokat?"
with PathEscape deep links (never QueryEscape) and class-driven copy;
data_paths: annotation with the Fork-3 asymmetry; system-owned beolvasas SMB
share refused server-side at handler AND store, button omitted in template.
Caught on the way: the sharing template's row struct was function-local, so
adding {{if .System}} would have 500'd every share row. ShareRow is now
package-level and the render test uses the handler's own type.
Tests 915 -> 949, all green. MinAgent unchanged.
agentapi.SmartSummary.ModelName mirrors the agent's model_name; the card row label
prefers the device model over the raw name/UUID, falling back to Name(+hint) on an
old agent. Additive. Test + red-proof (drop fallback -> A4 fails).
/ now 302s to /launcher (ONE canonical URL per page — launcher body never served at /);
the Vezérlőpult keeps /dashboard + its nav slot (operator ruling, reversing v0.163.0).
Nav: Indítópult active on /launcher, Vezérlőpult href=/dashboard active there. Post-login
default (/) and the topbar logo (/) flow through the redirect; login target unchanged.
Repointed 2 dashboard-card tests /→/dashboard. Corrected the stale vacation/agent-DOWN
note in CLAUDE.md (agent is up at 192.168.0.162:8443).
Tests: 302 target+status, /dashboard 200, nav hrefs/active; red-proof verified.
gofmt -w across the controller tree (46 files) so gofmt -l is empty — disarms the
formatting landmine where a targeted edit + accidental gofmt -w swept ~46 unrelated
files. Pure formatting: whitespace + gofmt's optional-semicolon removal in reflowed
inline closures. One doc comment reworded ('' -> 'the empty string') to avoid gofmt's
Go-1.19 doc-comment typographic substitition ('' -> curly quote) muddying its meaning.
No build/vet/test behavior change.
The agent defaults SMART to UNKNOWN on non-physical targets, so they showed as
spurious 'Nincs adat' rows. isPhysicalDisk now excludes those types (card + check).
Test strengthened: a PBS/LVM fixture with UNKNOWN SMART must still be excluded.
Consumes the agent v0.94.0 smart payload (MinAgent floor unchanged; feature-detect
by presence). One pure verdict fn agentapi.DiskVerdictFor shared by the dashboard
card and the 6h check. Card via a 60s /disks TTL cache (anti-smartctl-storm);
unreachable agent -> Nincs adat, page never blocks. disk-health-check (6h) emits
disk_health_degraded on a degradation only vs an in-memory baseline (first run
silent, recovery/UNKNOWN never notify, multi-attr -> one event). No global banner
(deliberate). Pairs with the hub allowlist bump.
Tests: verdict table (>=90 red-proof), notifier emit, check first-run-silent
(red-proof), degradation-once, recovery-silent, UNKNOWN-excluded, FAILING-critical,
nil-smart card, TTL cache.
ONE setting (window start W) drives every nightly leg at fixed, never-stored
offsets: DB dump at W, tier-2 at W+60m, off-box at W+105m (wrap-safe). Precedence
settings > controller.yaml db_dump_schedule > 02:30.
- scheduler.UpdateDaily: retime a daily job at runtime (no restart) via a per-job
buffered resched chan + a select case in runDailyJob.
- new pure package internal/backupwindow (LegTimes/GateWindow/EffectiveWindow).
- quiesce disk-tier window gate: scheduled cycles run only inside [W+2h,W+6h) with a
safety valve (age>cadence+24h runs regardless); manual TriggerNow never gated.
Backend.Due now also returns the backup age (from the agent's own /backup/due).
- backup page: Mentési időablak card (time input + derived leg/gate rows); POST
/backups/window validates -> saves -> UpdateDaily x3 -> flash.
Tests: 5 groups, all red-proofed. Agent/cadence//backup/due untouched.
.sidebar-logo was left-aligned; add auto side-margins so the 140px logo centers in
the header (desktop sidebar + mobile drawer share the element). Adds TestSidebarLogo_Centered.
Viktor pushed the text-outlined logo.svg to felhom.eu main (be9edb4) — wordmark is
now 17 real path glyphs. Swapped FelhomLogoSVG to it; stripped Inkscape's leftover
empty <text/> shells + font-* style leftovers on the paths via lxml DOM pass (glyphs
untouched, no text-to-path by CC), dropped editor namedview. Cleaned FelhomFaviconSVG
vestigial <text>. Both constants now free of <text>/font-family; viewBoxes unchanged;
palette + gradients preserved. TestLogoSVG_NoLiveText committed green.
Mobile nav was broken — the <=768px block predated the v0.146.0 accordion and
flattened .nav-links into a horizontal overflow-x strip, clipping the accordion's
nested sub-lists. Replaced with a sticky top bar + off-canvas left drawer that
reuses the vertical sidebar (accordion untouched), plus a no-js static fallback.
Removed the sidebar customer-name span (kept on login). Added ?v={{.Version}}
cache-bust to logo/favicon URLs (Cloudflare 4h edge-cache; 0.126.1 failure mode).
Part 4 (outlined-logo constant swap) gated out per §3a: live felhom.eu main still
serves a logo.svg with live <text>/font-family; constants unchanged.
5 new tests via the real layout/CSS render; nav_accordion invariants unchanged.
Feature-detected navigator.share opens the OS share sheet with title+text+URL only
(no QR files: — narrow support drops the URL when given file+URL). Hidden unless
supported; Link másolása stays the universal fallback and catches the non-cancel
rejection; AbortError is silent. Template JS + tests only. 2 red-proofs verified red.
Mint a 160-bit capability URL (/s/<token>) serving a standalone read-only guest
launcher: same tiles, opens apps in new tabs, no account, no admin session.
Information only, zero control — every privilege stays behind each app's own auth.
- /s/ pre-auth pass-through (after the claim gate) + session-CSRF exemption; guest
password POST carries its own pre-auth HMAC CSRF.
- Constant-time token match; empty stored token = disabled = byte-identical mux 404.
- Optional per-share password: separate bcrypt hash + own attempt map; signed cookie
= HMAC(token|passwordHash) keyed with web.session_secret, so rotate/change invalidates.
- Guest labels ride the v0.164.0 ruling; never expose internal state vocabulary.
- Token redacted in logs (/s/<redacted>); never in CHANGELOG/REPORT/CONTEXT.
- Admin modal: copy-link, QR (go-qrcode), set/clear password, rotate, disable.
- Tests: Groups A-G (14) + 3 red-proofs verified red.
A UI stop (Leallitas -> compose down -> StateStopped) is the user's own
action, not a fault, and must not raise the deadapp banner OR the
app_start_failed event. Filter at the single fix-3 derivation point:
extract scanDeployedAppRunStates's pure core to classifyRunStates and
change the down predicate to IsDownState(st.State) && st.State !=
StateStopped. Suppresses StateStopped from both the banner dead-list and
the notifier Down-set at once.
Rests on two invariants (recorded at the seam, README, CONTEXT):
I1 StopStack = compose down => zero containers => StateStopped
I2 P2 census: all catalog services unless-stopped => faults never rest
at stopped (they surface as exited/degraded).
IsDownState unchanged; out-of-band 'compose stop' (containers remain ->
exited) still alerts. Tests +4 (notify 3->4, main 4->7), both red-proofs
verified. No template/funcmap/notifier/counter/copy change.
(A) The launcher monogram bled through every transparent white logo — it rendered
unconditionally under the img. Now hidden by default (.launch-mono display:none),
revealed only when the img chain fails (onerror adds .launch-tile--noimg).
(B) The /static/app-placeholder.svg default reached only app_list_row. The four
other sanctioned app-logo onerror chains now match the canonical SVG->PNG->placeholder
grammar: backups_apps (aligned row), stacks (infra->infra-logo else placeholder),
app_info hero (screenshots still vanish), deploy (keeps .LogoURL/.LogoPNGURL).
Template/CSS only; no handler/funcmap change. 5 tests + 2 red-proofs.
New /launcher page: a grid of large tappable tiles, one per openable deployed
app (subdomain presence is the single openability criterion, shared with the
Megnyitás button via the extracted Server.subdomainMap helper). Colored tiles
(deterministic slug color or .felhom.yml brand_color), white glyph/monogram
fallback, target=_blank links for operational apps, greyed unclickable tiles for
stopped ones. First sidebar item; / stays the Vezérlőpult.
Universal app placeholder: new AppPlaceholderSVG served at
/static/app-placeholder.svg, now the default FallbackIcon on app_list_row so a
logo-less app shows a placeholder instead of visibility:hidden. Brand mark is
never an app placeholder.
New Metadata.BrandColor; new funcmap tileColor/initial. 10 new test functions +
4 red-proofs. No agent coupling; MinAgent unchanged.
admin
Claude Code