app_update_undone / app_update_held events (09 decision 15), on by
default and seeded once on existing boxes; R-606 update sentences as
key+args rendered per reader; R-646 startup applied-meta backfill for
apps current with the catalog; R-620 a disabled notifier WARNs once per
event type. Needs hub v0.120.0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The guarded update gains a folder copy of the app's named volumes, taken
after the pull where the app stops anyway (decision 19, chosen by the
2026-09-23 bake-off). On a failed health check the box undoes: every copy
validated by its finished-marker first, volumes refilled, definition and pin
from the job's own pre-update copies, the old version checked with the OLD
.felhom.yml probe. It holds only if the undo fails, and the hold sentence
says so and what state the data is in. Bind-mounted folders are never
touched.
- R-637 built; R-638/R-640/R-641 do not arise with a folder copy; R-639
(pre-update copies incl. .felhom.yml kept until the undo is over).
- journal phases copying/undoing with power-cut recovery.
- app.yaml last_update_undone + one line on the app page (hu/en).
- R-642: start/restart never answer "completed".
- Removal deletes kept undo copies.
MinAgent unchanged (0.131.0). Nine red-proofs in REPORT.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Caught by the live proof on 9202, not by a test. The v0.262.0 guard fired exactly right and
answered HTTP 500: router.go maps remove errors by grepping the error TEXT for "not deployed" /
"still running" / "not found" / "protected", and the busy sentence contains none of them.
A 500 tells the UI something broke; this is "wait a moment". Now a typed *stacks.RemoveBusyError
matched with errors.As and answered 409, carrying both the Hungarian bytes and the bundle key.
Its test asserts the sentence contains none of the words the text mapping greps for, so the type is
load-bearing rather than decorative. Red-proof seen failing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The controller self-updates daily at 04:30 by default, and after any hub report
once a floor sits above the box. That swap restarts the controller container.
The window proposed for automatic app updates is 02:30-05:00. It contains 04:30.
R-608 — a two-way lock, wired in main.go (stacks never imports selfupdate):
- stacks.Manager.AnyUpdating() -> Updater.SetAppUpdatingCheck, consulted in the
same three places as the existing backupRunning gate.
- Updater.IsUpdateRunning -> Manager.SetSelfUpdatingCheck; UpdatePreflight
refuses `self_updating`.
- MEASURED: the gap was narrower than assumed. The update's `backing-up` phase
already takes the backup single-flight, so that one phase was covered. The
other six were not, and `starting`/`verifying` are where data may have moved.
- The lock must NOT latch: a held app does not block the controller's own
updates, including the release that might fix the hold.
R-609 — the 409 carries `data.reason`, additively. transient (busy, updating,
deploying, migrating, self_updating) vs terminal (held, downgrade). Found while
writing the test: the router refuses a HELD app on its own line before the
preflight, so `held` would have been the one reason missing.
Five red-proofs, each seen to fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
MEASURED 2026-09-15 (BIGNIGHT Phase 6): privatebin updated 2.0.5 -> 2.0.6, catalog
reverted to 2.0.5, and the box read „Frissítés elérhető — ma" over an Update that
would have moved the pin BACKWARDS onto a possibly-migrated datadir.
- stacks.CatalogOrder: the comparison gains a fourth answer (Ahead) and moves out of
web, so the badge and UpdatePreflight cannot drift apart.
- The badge: ahead reads „Naprakész"/"Up to date", tag-ok, with a title saying why.
- The refusal: UpdatePreflight returns `downgrade` (409), born as a bundle key; the
API now renders update refusals through errText so it reaches English households.
- Ahead is narrow: every differing service must be orderable AND newer, else Behind.
- Ordering is util.Version.Compare behind a tag normaliser — no second comparator.
- Three red-proofs, each seen to fail.
R-589 was already fixed in v0.258.0; only its register row was stale.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
179 Hungarian sentences were built deep inside a package with fmt.Errorf and printed by
whoever caught them: too late to translate where they are shown, too early where they are
made. Every one now carries its key across that gap. ZERO Hungarian error literals remain.
util.MsgError does three things at once, each earned:
- Error() is the Hungarian, byte for byte, so every un-converted printer is unchanged;
- errors.Is answers for the kind AND for a wrapped cause (KindErrorf dropped the cause);
- an error ARGUMENT renders recursively, so "formázás sikertelen: %w" translates whole.
A foreign error — restic, docker, ssh, the stdlib — prints verbatim. It is not ours.
76 display sites go through errText, and TestNoErrErrorInPageOutput convicts any that do
not. memoryVerdict returns an error rather than a sentence, so the deploy's 409 and the
household's language come from one value; UpdateRefusal gained a Cause to carry it.
Plurals, one rule, stated once: a key with .one/.other takes its COUNT first. Not a
per-call-site flag — the producer somebody forgot would read "3 app is not running". The
guard caught a real key collision (alert.deadapp.one) the day the rule landed.
TWO DEFECTS FOUND IN MY OWN TOOLING, recorded rather than quietly fixed. The bulk converter
silently dropped multi-line concatenations, damaging 7 producers — and the parity gate could
not see it, because every surviving fragment WAS a real base literal while the CALL had lost
text; two behaviour tests caught it. And the counting script was case-sensitive, so it said
"0 left" while five remained.
MinAgent: 0.131.0 (unchanged). No hub release needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Slice 1 translated the dashboard's markup. The sentences the program BUILDS were still
Hungarian literals in Go, so an English household clicked an English button and was
answered in Hungarian. 226 of them move into the bundle here.
A flash was the hard part: it travels inside the redirect URL and is rendered by a
DIFFERENT request, so it now carries a bundle key plus its parameters. A link minted by
an older controller carries prose and is shown verbatim — never a raw key, never dropped.
Also converted: page data and view-model text, the internal/api JSON answers, the alert
banners (Alert.MessageKey, rendered on the way out of GetAlerts), 237 country names at
display, and the four page titles built around an app name (R-566 closed).
Hungarian is byte-identical, and that is measured rather than read:
scripts/i18n_go_parity.py freezes every Go literal at the base commit (7 467) and refuses
a key whose Hungarian is not that text, byte for byte. Three decoys, each seen to convict.
Its own first version filtered the capture through an ASCII-Hungarian word list and missed
seven real literals — the R-565 class. The filter is gone.
Nothing on the wire moved, and wire goldens now hold it there: the report's health
warnings and every notify event message stay Hungarian, because the hub MAILS the
controller's sentence when it has no entry of its own. Slice 3 (R-558) owns those.
MinAgent: 0.131.0 (unchanged). No hub release needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Every Hungarian sentence is byte-identical; each decision now reads a signal set where the message is
made. util.KindErrorf builds the same bytes fmt.Errorf did while carrying a sentinel for errors.Is.
- Deploy status (api/router.go): deployStatusFor() by kind — stacks.ErrAlreadyDeployed (409),
ErrRequiredField / ErrPathMissing / ErrNotEnoughMemory (400). The „kötelező" / „memória" /
"does not exist" / "already deployed" text chain is gone.
- Off-site failure class (backup/offbox.go): ErrOffsiteQuota replaces the „tárhelykeretet" match. The
restic/ssh signatures stay text matches on purpose — that output is not ours and is not translated.
- Alert placement (web/alerts.go): monitor.HealthReport carries WarningKinds parallel to Warnings;
the "not on a separate drive" warning is inline by KIND. The hub report is untouched (builder.go
copies Status/Issues/Warnings only) — pinned by a wire test.
- Stale off-site note (web/handlers.go): settings LastWarningKind + backup.OffboxWarnNoAppsSelected.
The text test survives ONLY for kind == "" (a box whose last run predates 0.251.0) and is removed
when R-570 closes; slice 2 must not translate that producer before then.
Tests (all red-proofed by restoring the pre-fix predicate — see the audit's redproofs.txt):
TestR553_Deploy_DecisionSurvivesWordingChange, TestR553_DeployHandlerUsesTheKind,
TestR553_DeployProducersCarryKindAndKeepTheirWords (through the real DeployStack),
TestR553_OffsiteQuota_{Decision,HeadLine}SurvivesWordingChange, TestR553_OffboxRunRecordsTheKind,
TestR553_StorageWarningsCarryKindsAndKeepTheirWords, TestR553_DiskWarningPlacementSurvivesWordingChange,
TestR553_HubReportWarningsAreUnchangedOnTheWire, TestR553_StaleNote*, TestR553_WarningKindIsPersistedAndCopied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
R-537 — the contents label is now PER TIER. One string computed from the app's
shape was rendered on all three tier rows; a Tier-1 unit has no file-copy step, so
for the four class-A apps it was claiming „Adatok" for files it does not hold.
R-538 — a unit restore REFUSES before anything is touched when the unit cannot
return the app's drive-side files, and names the route that can. It runs before the
stack is stopped because the measured harm included the app's own wastebasket going
unreachable, which still held every byte.
R-536 — „Alkalmazás telepítve" moved from the deploy's acceptance to its completion,
with app_deploy_started and app_deploy_failed as the honest pair.
Each fix red-proofed: seen failing with its own sentence, passing when restored.
Requires hub v0.116.0 for the two new event types. MinAgent unchanged (0.131.0).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
R-487: the local backup lists are keyed on the drives, not on what is
deployed — a removed app whose unit was kept is listed with the restore
that reinstalls it, the picker answers for it, and the restore opens the
unit where it sits. R-491: a removal clears the app's update hold.
R-490: /api/system/info reaches the API router and reads the default
storage path. R-489: volumes_removed is the real before/after difference,
[] when none. R-476: a Tier-2 copy is dated by its data, not its manifest.
R-456: the boot-orphan rule is pinned. Every fix red-proofed.
R-486 (P1): removing an app with its backups KEPT keeps its Tier-2 record,
so the second-drive restore is no longer refused over an intact mirror.
R-484: postgis/pgvector/timescaledb images are Postgres (logical dumps).
R-485: the backup card sizes the recovery unit and the mirror(s).
R-480: a held update's sentence leaves the card once the hold is lifted.
R-477: the update's off-site lookup is one snapshots call, no stats.
R-478: a copy older than this install's deploy does not count.
R-474: "delete backups" deletes the unit, the mirror(s) and the prefs.
Tests and red-proofs per row; evidence in felhom.eu
documentation/audits/v0240-2026-09-13/ and nightly-2026-09-13-adventurelog/.
Operator ruling 2026-09-13. The update precondition walks Tier 2, Tier 1
(own recovery unit, "helyi") and Tier 3 (off-site, 15 s bound; unreachable
counts as absent with a WARN) and leans on the first FRESH copy; the
backup_max_age rule applies to whichever tier is chosen. No copy anywhere:
back up first. Refused only when nothing exists and no backup can be taken.
RunAppBackupNow tolerates a Tier-2 failure (WARN) and marks the captured
unit proven current. The hold names the tier (második meghajtó / saját
meghajtó / távoli mentés) and the date; pre-v0.239.0 holds keep their text.
A successful off-site restore now lifts an update hold. The backups page
still uses Tier2UnitRestorePoint unchanged.
Scenarios G-M tested; red-proofs M, L, the tail and the off-site clear in
felhom.eu documentation/audits/rulings-r472-r475-2026-09-13/.
POST /api/stacks/{name}/update is now a guarded job answering 202:
cheap refusals (hold — R-439, busy, migration, deploying, memory via the
deploy's own memoryVerdict, a fixed 2 GB disk floor, and no restorable
Tier-2 copy) → backup-first when the proven copy is older than
update.backup_max_age (24h) → safety dump BEFORE the pin moves → pin →
pull (failure puts the pin back) → up → health (.felhom.yml check or 60 s
settle, update.health_timeout 5m). Not healthy → the app is stopped and
HELD (RestoreHold reason update_failed, same store and gate as R-379) and
the page names the backup to restore from; the pin stays. Success is only
ever update_phase=done after health (R-443). UpdateStack is deleted.
The restorable-unit predicate is EXTRACTED to backup.Tier2UnitRestorePoint
and shared with the backups page (row pinned unchanged). The copy is aged
by the last successful Tier-2 copy, not the manifest created_at — measured
on demo-hp that created_at moves only on definition changes.
Crash safety: update-journal.json before each phase; RecoverUpdates before
the boot sweep, ResumeInterruptedUpdates after the guards are wired.
Three unattended start paths ignored a hold and now honour it: the
drive-return gate (restart + boot recreate) and the nightly volume dump.
The nightly capture and Tier-2 run skip held apps so the restore point
survives. No automatic rollback — measured per-app; route back = restore.
Tests A–H across stacks/backup/api/web/cmd; six red-proofs seen to fail.
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Removal resolves the drive from the app's own app.yaml HDD_PATH (the 07 ~L437
rule), never the global cfg.Paths.HDDPath which no box sets. A data removal
that cannot be resolved, or whose drive is absent, is refused with a typed
RemoveRefusedError -> 409 + exact Hungarian sentence, before compose down, and
the app is kept. SSD app -> hdd_paths_removed: [] never null; missing folders
stated; backup-path refusals reach the response.
15 tests, two red-proofs run (pre-fix fallback -> C fails with err=nil and the
handler 200s; "no drive refuses" -> D fails).
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
R-379 and R-380 were one failure. Both ended with a half-restored database; the
only difference was whether it looked broken. Postgres emptied and crash-looped;
MariaDB applied part of the dump and reported health=healthy with a zero-row
schema-version table. Measured live on demo-hp 2026-08-22.
The undo copy was already taken and already good - proven by hand that day on
both engines. Nothing in the product could apply it. Now it does, with the same
ImportDump call, before any restart and inside the DB-only window.
The WHOLE undo set, matched on this run's stamp. writeSafetyDump returned one
path for an app with two databases; a rollback on that would restore one and
leave the other half-written.
When the rollback also fails the app is HELD STOPPED (operator ruling): a running
app on a half-written database lets the customer make the damage permanent. Every
start path refuses it - customer button, appstop Recover, boot sweep - via the
shared driveStartGate, checked ABOVE its driveless early return because these
apps have no drive. The marker is ended so nothing auto-restarts it. The row goes
red. Cleared with --clear-restore-hold, an operator CLI route.
--single-transaction is a belt on Postgres only; MariaDB DDL is not transactional
and that is why the rollback is the fix.
R-381: the engine's stderr stops reaching the customer (615 bytes on MariaDB, its
middle rows out of their own database) and starts reaching the operator log,
which never had it.
R-382: the summary log prints the volume count it already held.
Undo copies resolve to their own app, are marked IsUndo, and are capped at 3 per
app, pruned from the capture side. The reported render-as-an-app symptom did NOT
reproduce - the live page was read first and had zero occurrences.
Tests 1468 -> 1483. Eight red-proofs; ONE PASSED and is reported: the R-381
behavioural test injected below ImportDump. A guard at that layer now convicts.
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed
events reached the hub and TWO operator emails went out. The hub's operator
cooldown key is customerID:eventType(+tier) and that event carries `app` but no
`tier`, so the key held no app identifier: the first refused app took the hour's
slot and every other app's failure was discarded BEFORE anything was written
down, leaving no row on any channel.
The obvious fix — put `app` in the key — was ruled against: on a full disk it
produces one email per app, the volume problem wearing the correctness problem's
clothes.
internal/backup/runsummary.go: a per-run collector with exactly admissionSet's
lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the
end and only when something failed. A clean run emits nothing.
The per-app event stays and becomes the RECORD — the hub routes it record-only,
stored and logged every time, never competing for an email slot. The record and
the notification are now different things.
Deliberate skips (disconnected, decommissioned) are excluded: they have their
own alert, and a nightly email about an unplugged drive is one the operator
learns to ignore.
A manual run always reports: the digest carries a unique run_id the cooldown
cannot collapse. Someone pressing the button is actively trying to get a backup.
THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a
capture failure found between runs would be recorded and never notified — a new
silence introduced while closing one. That path emits a digest with NO run_id,
so the ordinary 1-hour cooldown caps it exactly as before while the mail now
lists every failing app instead of whichever was first.
A refusal is recorded ONCE, where the verdict is taken, not at the three legs
that consult it — R-181's contract is one verdict per app per run. Noting it per
leg listed one refused app three times and produced "2 of 1 apps failed". Found
by the digest's own test, not in review.
Silence is safe because the hub's deadline check raises expected_backup_missed
from report freshness, independently of any mail this box sends
(monitor/deadline.go:396,417). Confirmed, not assumed.
7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first
attempt — the AST test walked the backup package and not main.go; the test was
fixed and the mutation re-run rather than the pass recorded.
The box stops inferring the customer's intent from a container count and reads
what they actually asked for.
Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).
ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.
Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).
Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.
No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
This is D5's precondition and it is now met.
An app's namespace root IS its backup root: namespaceRoot returns a non-system
drive path as-is, so the recovery unit lands at <HDD_PATH>/backups/primary/<stack>/.
On a NAS that sits inside the share, which FileBrowser binds WHOLE — share root,
:rslave, download:true.
The bind was NOT narrowed, and establishing why inverted the fix. The share-root
:rslave bind is load-bearing (a 2026-07-22 probe proved an in-container access
through it wakes the idle automount trigger), and scoping is undefinable anyway:
apps on a share store at <share>/<app>, there is no userdata/ layer, and creating
one would write Felhom convention onto a customer's own NAS, which R-67 forbids.
So the browsing surface cannot be narrowed and the backup tree must never be
placed under it. Operator ruling: refuse the placement, keep the browse bind.
Tier 2 already refuses network targets for this reason (F-6C-1).
Nothing stranded: zero apps on network storage across all six hub customers
including Peti. R-67's browse capability is byte-identical.
FIVE surfaces, not the four the register named — settings.RefuseAsAppNamespace is
the single predicate. The deploy POST is the real boundary (it accepts any
caller-supplied HDD_PATH; DeployStack validates only os.Stat). Surface 4,
handleStorageDecommission mode=migrate, guarded only its SOURCE, so a whole
namespace could be decommissioned ONTO a NAS — that one is not in the register.
Fails closed: /mnt/felhom-drives holds both kinds, Kind exists only on a
registered path, so an unregistered path under that root refuses.
Supersedes README's "NAS backup locality — decision A" (v0.118.0).
9 tests, all non-effect (nil stackMgr, so a guard that misses panics rather than
passing). 4 red-proofs, each mutation asserted to have landed.
Suite rc=0, 27 packages, 0 FAIL. vet rc=0. Template + emoji gates OK.
${IMPORT_PATH} = <system namespace root>/userdata/import — ONE drop-zone per box,
on the system drive, injected at BOTH compose-env builders with NO per-drive
fallback (unresolvable leaves it unset so compose fails loudly rather than
quietly building a second, dead drop-zone).
Third BindRoot (RootImport) + Import list in BackupSpec, extended through
ValidateBackupSpec/ClassifyBinds. Load-bearing: a stale `userdata: import/<app>`
entry against the moved bind would be a WHOLE-BLOCK reject, taking the app's
mandatory hdd classification with it.
Exhaustive-root audit: resolveAbs/structuralGuard/ComputeCaptureSet/
ComputeFabBuckets now take importRoot explicitly (an import bind resolved
against hddPath would name a directory on the wrong drive); unresolvable is
refused loudly into Skipped. GetImportRoot added to both provider interfaces.
Catalog-derived skeleton: UserdataSkeleton() -> UserdataSkeletonCarry() +
BuildUserdataSkeleton(), SORTED. The carry-list makes zero-removals true by
construction (`documents` is in no catalog app but on both boxes) and is the
fresh-box floor. The sort is not tidiness: the naive map-order derivation
measured 20 distinct outputs from 20 identical runs, which with fbNeedsRecreate
is a fleet-wide FileBrowser restart loop.
One authoritative compose parser: ParseComposeUserdataMounts now delegates to
ParseComposeClassifiableBinds. Import root excluded from per-app migration.
Surfaces: FileBrowser /srv/beolvasas source; app-page "Hova tegyem a fajlokat?"
with PathEscape deep links (never QueryEscape) and class-driven copy;
data_paths: annotation with the Fork-3 asymmetry; system-owned beolvasas SMB
share refused server-side at handler AND store, button omitted in template.
Caught on the way: the sharing template's row struct was function-local, so
adding {{if .System}} would have 500'd every share row. ShareRow is now
package-level and the render test uses the handler's own type.
Tests 915 -> 949, all green. MinAgent unchanged.
The previous commit landed only the new test/badge files: a 'git stash' used
to compare REUSE.md ref-check output silently dropped the staged index, so
every modification to an existing file was left behind and that commit does
not build. This adds the metadata field, the predicates, the fail-closed
deploy gate, the catalog filter, the funcmap entries, the template edits and
the docs that those tests exercise.
The catalog knew only 'present' or 'gone', and 'gone' orphans every customer
already running the app. lifecycle: in .felhom.yml withdraws an app from new
installs without touching anyone running it.
Deploy gate is server-side and fail-closed, before any mutation, with the
ruled Hungarian refusal - hiding a button is not a gate. Unknown values fail
OPEN (available + one WARN), deliberately opposite, so a typo or a newer
catalog cannot pull a working app out of every customer's list.
Orphan detection never sees the field - a red-proof adds that filter and shows
the abandoned app immediately reading as an orphan.
Badge plumbing is generic (MetaBadge + meta_badge partial) so R-56's
difficulty labels drop in with no new markup.
Closes R-47. No new agent coupling — MinAgent stays 0.90.0.
The replay needs a running DB container, so both restore paths started the
WHOLE stack first, giving the application a window to rebuild the very schema
objects the dump was about to create. Measured live on 2026-07-19 (H4,
DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds
before the dump's CREATE INDEX, the replay aborted "already exists" under
ON_ERROR_STOP=1, and immich reported schema drift. The data survived only
because pg_dump emits COPY before CREATE INDEX.
Both paths now open a DB-ONLY window: only the stack's database service(s)
come up, the dump is replayed with the app still down, and the full start
runs only after the replay exits 0. Fail-closed: a dump with no identifiable
DB service refuses BEFORE the first mutation. Every exit from the window
still does a best-effort full start, so a failed restore never leaves a box
with a database and no application.
New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line
scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage
with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty
list — argument-less `up -d` is a full start); RedeployFromEnv split into
PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's
RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden
`up -d` inside the old name is what carried the defect on the local path.
19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed
effects, replay-failure bring-up, parser decoys, empty-list refusal); three
companion red-proofs run and reverted. 23/23 packages green.
Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0.
Task 2 of the backup-classification-redesign arc. Ships the referential-
coupling classification as DATA + PARSER + PURE CLASSIFIER, deliberately
inert — no backup tier changes behavior. Task 3 (tier policy engine) and
Task 4 (manual .fab UI) consume it.
- appbackup/classify.go: BackupSpec/BindSpec/ComposeBind/ClassifiedBind;
ClassifyBinds (SQ5 two-level default — explicit beats :ro; unlisted
writable→mandatory, unlisted :ro→excluded; nil spec→legacy/false);
ValidateBackupSpec (whole-block-reject on any defect, first defect named).
- stacks/classify_binds.go: ParseComposeClassifiableBinds — ${VAR}-relative
binds + :ro flag (NOT ParseComposeHDDMounts/ExportDataMounts, the traps).
- Metadata.Backup + LoadMetadata as the single validation choke point (bad
catalog block → nil + one ERROR → legacy, within one sync cycle).
- Manager.ClassifiedBinds + StackDataProvider.GetStackClassifiedBinds seam
(delegated by stackAdapter, nil-stubbed in every fake) — wired + tested
now so Task 3 consumes a tested seam.
INERT: full pre-existing suite green with zero test-logic edits. +14 tests;
red-proofs RP-1..RP-4 confirmed. The 13 catalog backup: blocks ship in the
same app-catalog change (this controller deploys first).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
Re-adjudicates F4: /backup/restore, /backup/tier2/restore, /backup/offbox/restore
blocked the HTTP request until completion, so through cloudflared's 100s cap a
customer got an error page while the restore succeeded (offbox worse — bounded
on r.Context(), canceling the SFTP restore mid-flight). Convert all three to the
offboxRun async shape: fast-path IsRunning refuse, background goroutine
(offbox ctx off r.Context() -> Background+30m), instant redirect. Add mutex-
guarded op-status (opstatus.go) + GET /api/backup/restore-status + a 3s-polling
backups.html banner (neutral running, red on failure). Restore single-flight
unchanged. Tests + red-proof (sync handler blocks indefinitely vs <500ms async).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The backups.html restore panel fetched /api/backup/snapshots (a restic-era route
that no longer existed), so the snapshot dropdown never populated and the
"Visszaállítás indítása" button could never enable — customers could not restore
anything from the UI (drill finding F1, DRILL-appdata-restore-2026-07-04).
- backup.Manager.ListRestorePoints: the keep-side restore has exactly ONE restore
point per app (the current recovery unit); time = newest artifact mtime among
manifest/db-dumps/volume-dumps; tier always 1 (Tier-2 copies are NOT restorable
via POST /backup/restore — never listed); drive_label from the storage registry,
empty for the SSD fallback.
- api: /backup/snapshots route + validStackParam guard (same semantics as
web.validStackName; traversal → 400, unknown stack → 404, no unit → ok+[]).
- Tests dispatch through Router.ServeHTTP (the bug WAS a missing route) + unit
tests for newest-mtime/label/empty semantics. Companion red-proof: hollow
always-[] implementation fails TestListRestorePoints_UnitOnDisk +
TestBackupSnapshots_UnitOnDisk (verified, reverted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
PushResponse.ConfigVersion from the report ACK; ConfigRefresher reconciles vs.
the last-applied version (settings.applied_config_version) and on a change calls
bootstrap.RefreshConfig (re-pull controller.yaml + re-merge local_api) then
GracefulSelfRestart. First-run records baseline (no restart); unchanged = no-op
(no storm); failed pull keeps config + retries. Companion to hub v0.26.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
- Geo settings save and manual geo sync now fire an out-of-band, non-blocking hub
report push (Router.reportPushNow seam, wired in main.go to BuildReport+Push in a
goroutine) so the hub reflects the new geo state / clears a stale last_sync_error
within seconds instead of after the next ~15-min cycle. Scope: geo handlers only.
- builder.go always populates report.GeoRestriction (Enabled=false, empty countries
when nil/disabled) via new buildGeoRestrictionReport helper, so the hub always
renders the geo section ("Inaktív" when off) instead of hiding it via omitempty.
- Tests: geo save success → push once; invalid country → no push (companion);
buildGeoRestrictionReport(nil) → non-nil disabled (companion vs old nil-omit).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /api/config/apply now takes effect via a graceful SELF-RESTART instead of
logging "restart needed" and leaving stale in-process singletons (the CF client
is built once at startup, so a rotated Cloudflare token never applied until a
manual LXC restart). Container is restart:unless-stopped, so a clean os.Exit(0)
auto-restarts with fresh config.
- New gracefulSelfRestart helper behind an injectable Restarter seam (Router.restart
+ SetRestarter) so the exit is unit-testable.
- configApply: no-op guard (byte-identical re-push → no write, no restart), else
write → 200 (flushed) → restart. Removed stale "restart needed" wording.
- Removed the dead OnConfigApplied hook (Phase-1-retired infra-backup push; the
self-restart reloads everything and a fresh report is pushed on startup).
- New POST /api/selfrestart (auth+CSRF via /api/ mount) + "Vezérlő újraindítása"
settings button: confirm → POST → poll GET / every 2s → reload.
- Tests: changed→restart once; identical→not called (companion); invalid→not called;
selfrestart→restart once.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drives are visible in-guest only at the STABLE /mnt/felhom-drives/<name>; the
registered path + HDD_PATH + FileBrowser source repoint there while agent calls
map back to raw /mnt/<name> (agentWhere). Enroll binds-under-parent before
register. Drive-absent GATE (planDriveGates + 30s driveGateLoop) stops/blocks
apps when a drive vanishes and auto-restarts on return; start-gate refuses start
when the drive is absent. H1 endpoints (disconnect/reconnect/restart-apps) routed
onto host-side ops. Non-hollow tests + companions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cgroup-only approach was a no-op on the demo: the controller container's OWN
cgroup is unlimited (the 2GB cap is on the LXC ancestor, hidden), and /proc has no
lxcfs, so it kept reporting the host's 16GB. The Docker daemon runs IN the LXC, so
'docker info' MemTotal reports the guest's real cap (2048MB) — now the authoritative
source (cgroup limit preferred when present, e.g. non-nested). The deploy memory
guard now uses the controller's committed-app memory (sum of running mem requests)
for 'used' — accurate and cheap — instead of host /proc RSS (unobservable per guest;
would make the guard never/always fire). /api/system/info reports the guest cap as
total and committed memory as used. Tests: cgroup-limit path, docker-info fallback
(nested case), GuestMemTotalMB fallback (dockerMemTotalFn stub).
The Hub config-apply handler wrote controller.yaml 0644; it holds cf_api_token,
cf_tunnel_token and hub api_key in plaintext. New writeConfig0600 helper writes
0600 atomically (tmp+rename, bind-mount fallback) and chmods to enforce 0600 even
when the file pre-existed 0644 (os.WriteFile doesn't chmod existing files).
Test asserts mode 0600 (Linux; skipped on Windows). Setup path already used 0600.
F4: GET /api/stacks/rescan fell through to GET /stacks/{name} → misleading
'stack not found: rescan'. Now returns 405 + Allow: POST.
F6: the deploy POST returns before compose/health complete (async; UI polls).
Message changed 'Stack X deployed' → 'Telepítés elindítva…' and status 200→202
Accepted, so API/script consumers aren't told a deploy finished when it hasn't.
UI checks data.ok (not HTTP status), so 202 is safe.
Reserved-buffer headroom guard on the Docker-data volume (system/dockervol.go,
max(5GB,10%)); deploy-time hard gate refuses (HTTP 507) when below the buffer
(api/router.go); deploy page warns + disables the button (deploy.html); runtime
disk monitor confirmed to watch the Docker volume above the buffer. Log rotation
baked into the golden (agent side). Phase 1 = felhom-agent v0.29.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tier 2 rsync-mirrors each HDD app's recovery unit + appdata to a DIFFERENT physical
disk (the only off-drive protection bind-mounted userdata can get; PBS can't reach it).
Auto-enabled, auto-target: prefer another registered drive (different physical disk via
system.SamePhysicalDevice), else the internal SSD for SMALL units only — with a
size-aware headroom guard that REFUSES rather than fill the ~8G guest rootfs, recording
an honest "needs 2nd HDD" status. Status persisted via the surviving CrossDriveBackup;
"2. mentés" UI card now populated. Daily tier2-backup job + POST /api/backup/tier2.
- backup/tier2.go (engine+selection+headroom), tier2_test.go (headroom arithmetic)
- system.SamePhysicalDevice (linux Stat_t.Dev + stub)
- handlers.go Tier2 UI population + tier2DestLabel; backups.html honest no-target reason
- fixed stale TestBackupCopiesOnPath (old felhom-data layout -> in-guest layout)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Backups page: whole-guest backup shown as real DR — target label "Biztonsági szerver –
külön hardver (PBS)"; app-data "Távoli mentés" card now reflects the PBS offsite tier
(guestBackupView.Offsite) instead of "nincs beállítva".
- Model-A double-nest fix: appbackup path helpers take a felhom-data NAMESPACE ROOT (no
internal felhom-data join); backup.Manager.namespaceRoot/AppNamespaceRoot resolve
HDD-vs-systemDataPath provenance so a drive-resident app's backups land single-nested
(<drive>/backups/... on the guest = <drive>/felhom-data/backups/... on the host) instead
of .../felhom-data/felhom-data/.... Writes, deletion (GetStackBackupData/RemoveStack/
ProtectedHDDPaths), wipe-warning scan, and export updated coherently; legacy double-nest
dirs kept protected. New appbackup test asserts no doubled segment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add Docker named volume backup to Tier 1 (dump to tar, include in restic)
and Tier 2 (copy tars to rsync mirror _volumes/ dir)
- Fix volume name resolution: use project-prefixed names (mealie_mealie_data)
- Fix double Tier 1 in restore dropdown: filter snapshots by app's home drive
- Add Tier 2 restore: RestoreAppFromTier2() restores from rsync mirror
- Show Tier 2 entry in restore dropdown when cross-drive backup succeeded
- Add .fab import link in restore section
- Volume-aware restore type banners and backup content labels
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add structured operational logging at INFO, WARN, and ERROR levels to
every controller module. Standardize custom prefixes ([GEO], [SCHED],
[SYNC]) to use [INFO/WARN/ERROR] [module] format. Fix misleveled logs
(WARN->ERROR for data loss scenarios, WARN->INFO for routine operations).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add detailed [DEBUG] logging to every controller module when
logging.level is set to "debug". Each module with stateful debug
uses SetDebug(bool) wired from main.go. Covers stacks, backup,
cloudflare, integrations, system, monitor, settings, scheduler,
web handlers, storage, metrics, API, selfupdate, and assets.
Also includes the app export/import (.fab bundles) feature from
v0.32.0 and its debug page integration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Generic integration system for connecting deployed apps via toggle UI.
First handlers: OnlyOffice→FileBrowser (config.yaml patch) and
OnlyOffice→Nextcloud (occ CLI). Lifecycle hooks auto-suspend on
stop and re-apply on start.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bug fixes:
- Add applyEnvOverrides to LoadFromBytes (M05)
- Set state=failed on compose-up failure in selfupdate (M16)
- Clamp usableMB to min 0 in memory check (M22)
- Remove "manual" schedule from triggerAllCrossBackups (M23)
- Add mmcblk device handling for partition paths (M21)
- Fix stripPartition for mmcblk devices (L25)
- Fix TruncateStr for UTF-8 and negative maxLen (L05/L06)
- Fix AllDone to return false for empty restore plans (L14)
- Fix PushOnce to return actual errors (L39)
- Restore pending events on save failure in DrainPendingEvents (M03)
- Add duplicate check in AddStoragePath (M04)
- Call CleanupTempMounts after drive scan (H13)
- Log SetStep save errors (M25)
Hardening:
- Guard scheduler Start() against double-start (M14)
- Acquire mutex in scheduler Stop() before reading cancel (L24)
- Cap log lines parameter to 10000 (L31)
- Require POST for logout (L32)
- Use sync.Once for Server.Close() (L49)
- Panic on crypto/rand.Read failure in setup CSRF (L40)
- Validate Bearer token against Hub API key in CSRF (H16 fix)
- Replace custom hasPrefix with strings.HasPrefix (L13)
- Replace simpleHash with crc32.ChecksumIEEE (L48)
Cleanup:
- Remove dead imageName function (L02)
- Remove dead detectHostIPViaRoute function (L03)
- Rename shadowed copy variable to cp (L07)
- Copy DefaultEnabledEvents in GetNotificationPrefs early return (L09)
- Update BUGHUNT.md with comprehensive audit results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add country-based access control managed through the Settings page.
Global allow-list with per-app overrides, searchable country selector,
automatic sync to Cloudflare WAF on settings change / deploy / remove,
plus periodic 6-hour verification.
New package: internal/cloudflare/ (client, zone, waf, countries, geosync)
New API: /api/geo/* (6 endpoints) + /api/stacks/{name}/geo/override
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy page, pre-start check, and deploy validation now use actual
/proc/meminfo usage instead of declared mem_request sums. New
GetMemoryMB() helper for lightweight real-time memory reads. Monitoring
page gains a stacked memory distribution bar showing per-container
usage, OS overhead, and free memory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Debug-mode-only dashboard (/debug) with 8 collapsible sections:
system diagnostics, notification testing, backup triggers, storage
simulation, hub & connectivity, self-update dry-run, DR/setup wizard,
and in-memory log viewer. Migrates debug dump from API router to web
server. Adds ring buffer log capture, storage disconnect simulation,
event history tracking, and cross-drive/self-update test methods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add internal/assets package that downloads and caches app assets from
Hub API with SHA-256 change detection. Assets resolve from synced cache
first, falling back to baked-in directory. Daily sync schedule +
on-demand POST /api/assets/sync endpoint.
Config: assets.sync_enabled + assets.sync_schedule (default 05:00)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
admin