72368654e42114864a49063a8d42811d30b626fc
136 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 admin
|
72368654e4 |
R-241 part 5: escalating reminders, and operator levers for a running countdown
REMINDERS (SEC 2.3). The offer epoch now stamps when it began, and the undecided reminder escalates in EMPHASIS at 1, 3, 7 and 14 days. THE READING IS STATED BECAUSE THE SPEC IS AMBIGUOUS, and it is written into the code where it can be corrected. For an ABANDONING box, 5/3/1 are unambiguously days REMAINING before a deletion. An undecided box has no deadline - nothing counts down to anything, because SEC 7.5 deliberately does NOT auto-abandon - so 14/7/3/1 cannot be "remaining" and are taken as days ELAPSED, with the wording firming up rather than the bar appearing and disappearing. If the operator meant something else, one function changes. The stamp is re-set on every entry into the offered state, so a box that settles and is later rebuilt starts its ladder again instead of inheriting an old one. OPERATOR LEVERS (SEC 7.5). --abandon-status, --abandon-extend=N and --abandon-stop on the controller CLI, beside the existing operator subcommands. They exist because the path that ACTUALLY happens is the customer telephoning, and support needs something to press. They live on the CLI and not in the customer UI deliberately: extending a deletion the customer asked for is an operator judgement, and a customer who wants it stopped already has the self-service route - they recover with their code, which cancels it. BOTH REFUSE RATHER THAN NO-OP, in two situations: when no countdown is running, and when the store has already been deleted. A silent success is the thing an operator most easily mistakes for "handled" - they would tell the customer their data was safe when it is gone. Pinned by two tests. --abandon-extend counts from NOW, not from the old due date, and a test proves the old date passes without deleting anything. Green: go build, go vet, go test ./... all pass; controller gates OK. |
||
|
admin
|
a5d90ff801 |
R-241 part 3: abandoning starts a 14-day countdown that ends the question
Until now "set aside" renamed the remote store and touched neither the escrow
nor the key, so the hub went on holding a sealed package for a key the box no
longer used. Shape (c) compares those two, finds them different, and offers
recovery - correctly, and for ever. A customer who had already said "I do not
want the old data" would be asked again at every login.
The operator's ruling is that the answer is NOT a "they decided" flag: fix the
state, do not remember that it is wrong. So the decision starts a countdown,
at the end of which the set-aside store and the sealed package that protects
it are removed TOGETHER. Afterwards shape (c) has nothing to compare and the
offer falls silent on its own - because the state is right, not because
something remembers it once was not.
THE GRACE IS REAL. The recovery offer stays reachable for the whole 14 days;
that is the change-of-mind path, and a grace in which recovery is impossible
would be decorative.
BOTH HALVES OR NEITHER. Removing only the store leaves a package that opens
nothing; removing only the package leaves ciphertext nobody can ever decrypt.
The two cannot be atomic across two machines, so it is a two-phase commit:
delete the store, record a durable marker, and keep DECLARING
offsite.abandon_purge_requested until the hub's ACK stops reporting a
superseded package. A crash between the halves re-declares on the next sweep;
it never leaves the pair half-removed and silent.
HUB HALF - SEC 8.2 ANSWERED: yes, the hub was needed, and only for this.
store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is
ever undone, and it never touches host_escrow (the package covering the key
the box uses now). The handler acts on the DECLARATION, never an inference,
and is placed immediately BEFORE the ACK is built - so
GetEscrowStatusForCustomer reads the effect and the SAME response closes the
box's two-phase commit. No second round-trip and no window where the box
thinks it is still owed. felhom-agent was NOT touched.
The countdown starts in ResetOrphanedRepo, NOT in the shared helper: the
helper is also the unclaimed auto-reset path, where nobody decided anything,
and an as-delivered box tidying a stranger's leftover store must not get a
customer's deletion clock. Pinned by a test.
Cancellation is wired into the recovery unlock, BEFORE the tier-up and the
listing - those can fail, and a countdown surviving a successful unlock
because a later step errored would delete the history the customer just
proved they can open.
The sweep is a Daily job at 05:10, not on the backup leg: it must run on a box
whose tier is not configured for runs. Quiet by construction on every box with
no countdown, and that silence is asserted.
Tests (all clock-injected; SEC 7.4 forbids shortening a live timer):
Scenario E (aside + package kept + countdown + offer still reachable, and
NOTHING deleted), Scenario F (both halves, the declaration repeating, the
close-out), Scenario G (cancel, path still nameable, no later deletion),
plus: not closed out while the package remains, a transport failure leaves the
countdown due and retrying, the no-op sweep issues zero remote commands, and
the unclaimed auto-reset starts no countdown.
RED-PROOFS, each with the mutation confirmed present in the file first:
F1) store deletion skipped -> Scenario F FAILS (no rm issued)
F2) declaration dropped from the report -> Scenario F FAILS (the hub is
never asked; the package would outlive the store for ever)
G) CancelAbandon made a no-op -> Scenario G FAILS (uncancellable countdown)
Green: controller and hub both build, vet and test clean; controller gates OK.
NOTHING WAS DELETED ANYWHERE - the terminal step has only ever run against
in-test fakes.
|
||
|
admin
|
a491abef6c |
R-241 part 2: the comparison the box already makes becomes the thing that offers recovery
THE FACT WAS COMPUTED EVERY CYCLE AND KEPT NOWHERE. EscrowAutoConfirmer.Reconcile
has compared the hub's restic_pw_sha256 against the local key on every ACK since
SLICE 3. On the final-walk venue it logged, at 03:28:03Z and thirty-five minutes
before the customer looked, "the hub's escrow blob does not cover the CURRENT repo
password (hub hash 30ef574f != local 9b4a9a9d)" - and dropped it. The recovery
screen, evaluating in the same process, went on asking a question that could not
see it.
Now persisted: settings.HubEscrowKeySHA256 + HubEscrowKeyCheckedAt, recorded
UNCONDITIONALLY in Reconcile beside RecordPresence and RecordSuperseded - same
place, same reason: the box that needs it most is the rebuilt one with no target,
on which every gate below returns early.
OffsiteRecoveryOffer gains SHAPE (c): the hub holds a package for a key OTHER than
the one we are using. (a) and (b) are both proxies for that question and both have
now been wrong in opposite directions - (a) goes false the moment anything mints,
(b) is unreachable while the escrow is pending.
SEC 7.2, decided deliberately and stated in the code:
- a KNOWN DIFFERENCE offers, however old the reading. Age is not gated on. Both
sides are local; only the hub's half can be stale, and what the hub holds does
not change without a ceremony THIS box runs, which refreshes the hash on the
next ACK. Gating on age would make a box offline from the hub silently stop
offering - the exact failure this session removes. CheckedAt is persisted for
diagnosis, not as a gate.
- an ABSENT hash falls back to (a)/(b) and does NOT offer. "" is the hub
positively saying its package seals no repository password (legacy hash-less
escrow). Nothing to compare, and offering would put a permanent screen in
front of every legacy box.
The write damper: CheckedAt refreshes on every ack carrying a hash, but a save is
skipped when both the hash and the UTC day are unchanged, so an idle box does not
rewrite settings.json every fifteen minutes. It records WHEN WE LAST HEARD, not
when it last changed - the R-100 distinction.
Tests: Scenario C (a differing key offers, with both proxies asserted false first),
Scenario D (a matching key offers nothing), fact 1 still required, shape (a) still
works, and both SEC 7.2 halves.
RED-PROOFS, each with the mutation confirmed present in the file first:
D) hubHash != localHash conjunct dropped -> Scenario D FAILS (a healthy box
offered recovery forever); Scenario C still passes
WIRING) RecordEscrowKeyHash removed from the EscrowAutoConfirmer literal in
main.go -> TestMainWiresRecordEscrowKeyHash FAILS. This is the ships-inert
shape: unwired, everything compiles, every test in the package passes, the
auto-confirm still works, and shape (c) reads an empty hash forever.
Green: go build, go vet, go test ./... all pass.
|
||
|
admin
|
9dc26459ea |
v0.203.0: the box collects what the hub staged for it (R-218 consume half) + R-220's message
gates / gates (push) Successful in 10s
R-218's declaration half shipped in v0.201.0 and works. Its consume half never existed. Reconcile ran exactly twice per process — at start-up and when the recovery screen drives it — and BOTH fire before the hub has anything staged, because the hub stages in RESPONSE to the declaration those runs precede. Measured on the R-201 re-walk: unlock reconcile 11:43:07, hub staged 11:44:57 saying 'next cycle', a full report cycle ran 11:55:46, still unconsumed at 12:06. A guest command line applied it in 18 seconds — everything correct except the trigger. Bridge.RetryIfDeclared re-runs the SAME reconcile on a 5-minute tick, driven from the box's own published declaration (OffboxReportStatus().State) — the very statement the hub acts on, so the two cannot disagree. Poll, not an ACK flag, decided on the promise: the no-target message says 'amint megvannak' (no deadline) and the card says 'within a day'. Five minutes is inside both by a wide margin and needs no hub change. It stops by construction — a healthy box does no work and logs nothing — and the settle gate is deliberately kept via ReconcileWhenSettled. The marker was investigated and left alone: applied_marker lives in the guest's DataDir, which a rebuild destroys, so it cannot suppress a legitimate re-run. R-220's customer half: the refusal no longer tells the customer to choose from a list that may be empty. It names the rebuild, points at the Meghajtók page, and promises no outcome. Red-proofs: remove the retry -> credential uncollected (the dead end reproduced); drop the stop condition -> a healthy box hammers the hub; call Reconcile instead of ReconcileWhenSettled -> settle gate bypassed; restore the old sentence -> the impossible action returns. 28 packages ok, vet clean, all controller gates OK. |
||
|
admin
|
a3499d1807 |
v0.201.0 — a correct recovery code is never called wrong again (CAMPAIGN-11) — MinAgent 0.125.0
gates / gates (push) Successful in 9s
R-216: the offsite key recovery is a coupled feature and now says so. featureProbes +
featureMinAgent 0.125.0 + a Supports gate at the unlock entry point, FAILING CLOSED — an
agent that cannot answer is named as such instead of the customer's code being blamed.
Measured live: a 404 from agent 0.120.0 came back as "we did not accept your recovery
code, check that all ten words", in 0.134 s, against a perfect code.
R-218: delete the repo-password short-circuit in needsOffsiteCredential. The declaration
stops when the TIER WORKS, not when a key exists — installing a key is the recovery
screen's whole job, so succeeding at recovery was switching off the mechanism that would
have delivered the coordinates to use it.
R-219: the unlock finishes the job — place the key, bring the tier up, then list. Without
it the promised listing could never render on the shape the screen exists for.
R-217: an unreadable store no longer claims to have opened with unattributable content
(the OffsiteInventory{} zero value). Opened / empty / unreadable are three states.
R-222: a code that is right about a RETAINED earlier package is named, not blamed. States
what the hub knows and promises nothing — no read path exists.
R-215: GET /recovery is gated on the same predicate as the interception.
Five red-proofs, each demonstrated failing and restored.
|
||
|
admin
|
636c51e542 |
R-193: the recovery screen — unlocking, and only unlocking (v0.200.0)
A customer whose machine was rebuilt had everything needed to get their data back and no way to find out: the only route was a command line. This is the screen that closes that. IT UNLOCKS, AND ONLY UNLOCKS (operator ruling). It explains, takes the recovery code, opens the repository and shows what is in there — apps, dates, sizes. It restores nothing: restore is already per-app and lives in the backups area, and a screen that unlocks and then offers to overwrite is two decisions wearing one button. ONE CORE, TWO CALLERS. RecoverInstallCore is split out of RecoverAndInstall; the CLI wrapper keeps its exit codes and printed lines byte-identical, and the handler drives the same function. Two implementations of the one operation that can permanently lose a customer's data would drift, and only one would be tested. Asserted from source on both sides by AST. THREE WAYS OUT, none a dismiss button: recover; 'most nem' (the full page stops interrupting, the backups-area entry point stays PERMANENTLY, bound to the offer and never to the postpone flag); and 'I do not want the old data' — confirmed TWICE and reaching the SHIPPED move-aside, which sets aside and never deletes. THE CODE IS HANDLED NO MORE LOOSELY THAN ON THE COMMAND LINE: POST body only, never logged, never persisted, never echoed, cleared on every path, no-store, autocomplete off. No lockout — the code is a ten-word phrase, and locking a customer out of their own data for a typo is worse than anything it prevents. TWO DEFECTS THE TESTS CAUGHT, both fixed: an UNCLAIMED (legacy-open) box would have been shown the page, because RequireAuth passes such a box through; and the inventory nil-dereferenced when no off-site target was configured, which is exactly the pristine rebuilt shape. |
||
|
admin
|
1214bae0a2 |
R-204 item 4 (box half): a rebuilt box DECLARES that it needs a credential (v0.199.0)
An absent off-site object has four meanings — never configured, mid-restart, a transient config read failure, and rebuilt-and-stranded — and the hub cannot tell them apart. The box can, from two local facts it holds with certainty, so it says so instead of leaving the hub to deduce it from a silence (operator ruling). The ACK's identity_blob_present is now recorded on EVERY ACK, before the gates that used to discard it: on a box with no off-site target the auto-confirm returns immediately, which is exactly a rebuilt box, so the one fact distinguishing it from a box that never had off-site backups was thrown away every cycle. The declaration needs BOTH halves — a fresh data area AND a hub-held recovery package. Freshness alone is a box that never had off-site backups; dropping that condition makes the whole fleet ask for credentials, which is what the Scenario B test exists to catch. The object carries enabled:false and zero sizes, which is what makes it inert to the hub's existing fill and staleness checkers and to a pre-upgrade hub. A configured box's JSON is byte-identical to v0.198.0's. |
||
|
admin
|
58c703bd44 |
R-203 Part 2: a run that missed a MANDATORY directory is not a successful run (v0.197.0)
gates / gates (push) Successful in 8s
The gap was already detected and warned about, in Hungarian, naming the app and the folders -- that warning is what stopped the R-201 drill. The defect was that the run still reported `ok` beside it, and a warning standing beside a success is read as a success. last_status gains "incomplete": minted, because "ok" | "error" | "running" had nothing meaning "it ran, and this app is not fully protected". NOT "error" -- the rest of the run worked and what was captured is real, so SnapshotCount and the LastSuccess anchor still record it. Half a backup is not no backup. The gaps are now recorded STRUCTURALLY (offboxRunResult.mandatoryGaps), not only as prose, so the verdict has something to act on. It reaches the operator through the EXISTING per-run digest (backup_run_failures) rather than a new event type -- a new type is a two-repo change and the hub drops anything outside allowedEventTypes. The stat-filter gains the ClassMandatory check Tier 2 already had. It is a NO-OP today (TierOffsite admits mandatory only), so no customer-visible warning disappears -- demonstrated by widening the tier filter alone and watching the check hold the line. ANTICIPATED: calibre-web on demo-hp has exactly this gap, so its off-site status becomes incomplete the moment this ships. That is correct and is the point. Red-proofs: my first Scenario-C proof PASSED because the test only reached offboxCaptureSet while the mutation lives in runOffboxInternal -- a mutation the test cannot observe is not a red-proof, and the fix was the test. The run-level test now fails under both mutations (unreachable gap recording; unconditional ok). |
||
|
admin
|
a96c3d9473 |
R-203: the export-mount resolver takes the namespace root too (its own commit)
gates / gates (push) Successful in 9s
ExportDataMounts lives in delete.go, which reads as a destructive path. IT IS NOT: its single production caller is the .fab export adapter, and nothing deletes based on its result. The delete path's own guard, ProtectedHDDPaths, is layout-agnostic by construction -- it protects BOTH <hdd>/... and <hdd>/felhom-data/... -- so deletion was never affected by the namespace-root defect. That scope note is now in the function's doc comment, because the file placement will mislead the next reader exactly as it misled the spec for this change. Separated into its own commit anyway, so a change to a function whose filename says "delete" is reviewable on its own. An empty nsRoot falls back to hddPath -- the pre-R-203 shape -- so any caller not yet updated keeps working on enrolled drives. Tests cover both drive kinds and assert the NEGATIVE: no emitted path lies outside the app's own data roots. Red-proof: leaving the site bare fails the system-drive row, emitting /mnt/sys_drive/userdata where the canonical root is /mnt/sys_drive/felhom-data/userdata. |
||
|
admin
|
73efb091d9 |
R-203: the app and its backup look in the same directory — one resolver, every caller
gates / gates (push) Successful in 9s
appbackup's path helpers take a NAMESPACE ROOT. Five call sites passed a bare DRIVE path.
On an enrolled drive the two coincide, so nothing showed; on the system-data fallback they
differ by exactly the felhom-data segment, and the app then bound a directory the off-site
capture set never looked at -- while the run reported ok. Measured live on demo-hp: the app
wrote to /mnt/sys_drive/userdata/media/books, the capture set looked for
/mnt/sys_drive/felhom-data/userdata/media/books.
THE RULE NOW HAS ONE EXPRESSION. appbackup.NamespaceRootFor / IsEnrolledDrive encode the
drive-kind comparison; backup.Manager.namespaceRoot and stacks.Manager.inGuest delegate to
it. There were already TWO copies and they differed -- the backup package's compared without
filepath.Clean, the stacks package's with it, so a trailing slash from config would have
flipped the mode in one and not the other.
Sites routed through it:
- stacks/deploy.go withPathVars -> ${USERDATA_PATH} (the live defect)
- appexport/fabplan.go + export.go (via a new provider method)
- web/handlers.go FileBrowser mounts (latent: the system drive is
deliberately never a registered StoragePath, so this is the identity today)
ComputeFabBuckets now receives the namespace root, which is what ComputeCaptureSet has always
received -- so the export's classified paths and the backup's capture set describe the same
directories by construction instead of by coincidence.
Tests are table-driven over BOTH drive kinds, because this survived by being invisible on the
kind that already worked. Red-proofs observed: restoring the bare-path call fails the
system-drive row with the two paths differing by /felhom-data; inverting the drive-kind
comparison fails every enrolled row.
|
||
|
admin
|
1b1366bb6e |
controller v0.196.0: the recovered key installs itself (R-200 plumbing half) -- MinAgent 0.125.0
gates / gates (push) Successful in 8s
--recover-offsite-install is the sibling of --recover-offsite-check: same fetch/unseal path through the agent, same STDIN discipline for R, but it PLACES the recovered repository password via InjectOffboxPassword so a rebuilt box reopens the history it inherited. Doing this by hand would put the offsite DATA key through a terminal, a clipboard and shell history. In-process the value goes agent -> this process -> the 0600 file and is rendered nowhere. The confirmation is a SECOND invocation: without --confirm-install it prints both hashes and writes nothing, so the operator sees the comparison before any write is possible. Three outcomes, named distinctly: installed (no local password -- the rebuilt-box shape), unchanged (identical key already present, nothing written), refused (a DIFFERENT key present; installing would clobber the key the current repository is encrypted under, and no force option is offered). Exit 2 for the refusal, distinct from 1 for a failed step. Red-proof: removing the confirmation gate makes the dry run write, failing the test. The R-persistence test carries a positive control -- a planted copy is found, then removed and not found -- because an absence check is worth only what its sensitivity is. |
||
|
admin
|
9640e51321 |
controller v0.195.0: prove the offsite key comes back (R-200 plumbing half) -- MinAgent 0.125.0
gates / gates (push) Successful in 10s
--recover-offsite-check is a docker exec diagnostic in the shape of --print-reset-code: it reads the customer's recovery code from STDIN, asks the agent to fetch this host's sealed bundle and open it, and reports whether the recovered key matches the one on disk BY SHA256. Two hashes and a verdict; never a password, never R, never a blob. R comes from stdin and not a flag because a flag value is visible in ps, in shell history, in a container's command line and in any transcript of the session that ran it. IT COMPARES; IT DOES NOT INSTALL. The recovered password is never written to offbox/repo_password -- installing changes a live box on a path nobody has walked, and that link is next session's, with the drill around it. A test asserts the data dir is byte-unchanged after a check; its red-proof (adding the install call) fails it. Exit codes: 0 match, 2 clean MISMATCH, 1 a step failed -- "it failed" and "it worked and disagreed" must never share a status. A box with no local password reports distinctly: that is the rebuilt-box shape, where the next step is to install rather than compare. Nothing customer-reachable ships here: no card, no form, no preview. |
||
|
admin
|
88897a224e |
v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)
gates / gates (push) Successful in 8s
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed events reached the hub and TWO operator emails went out. The hub's operator cooldown key is customerID:eventType(+tier) and that event carries `app` but no `tier`, so the key held no app identifier: the first refused app took the hour's slot and every other app's failure was discarded BEFORE anything was written down, leaving no row on any channel. The obvious fix — put `app` in the key — was ruled against: on a full disk it produces one email per app, the volume problem wearing the correctness problem's clothes. internal/backup/runsummary.go: a per-run collector with exactly admissionSet's lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the end and only when something failed. A clean run emits nothing. The per-app event stays and becomes the RECORD — the hub routes it record-only, stored and logged every time, never competing for an email slot. The record and the notification are now different things. Deliberate skips (disconnected, decommissioned) are excluded: they have their own alert, and a nightly email about an unplugged drive is one the operator learns to ignore. A manual run always reports: the digest carries a unique run_id the cooldown cannot collapse. Someone pressing the button is actively trying to get a backup. THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a capture failure found between runs would be recorded and never notified — a new silence introduced while closing one. That path emits a digest with NO run_id, so the ordinary 1-hour cooldown caps it exactly as before while the mail now lists every failing app instead of whichever was first. A refusal is recorded ONCE, where the verdict is taken, not at the three legs that consult it — R-181's contract is one verdict per app per run. Noting it per leg listed one refused app three times and produced "2 of 1 apps failed". Found by the digest's own test, not in review. Silence is safe because the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail this box sends (monitor/deadline.go:396,417). Confirmed, not assumed. 7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first attempt — the AST test walked the backup package and not main.go; the test was fixed and the mutation re-run rather than the pass recorded. |
||
|
admin
|
5adae4dad9 |
v0.191.1 — the fill check also runs at startup (R-167)
gates / gates (push) Successful in 9s
Found live on 9201: neither sched.Daily nor sched.Every fires on registration, so a box booting with a filesystem already over the line would stay silent for up to 24h — the R-100 shape, and the same gap the hub's own checkers avoid by leaving already-breached keys unseeded at init. The watcher now runs once 90s after startup as well. Safe because the check is edge-triggered against persisted state: an already-warned filesystem stays silent. The delay lets mounts settle so a drive still returning reads as unreadable and is skipped rather than warned about. Pinned by an AST assertion — the schedule registration alone no longer satisfies the test. |
||
|
admin
|
cf48214f6c |
v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored. |
||
|
admin
|
dcc3363d2f |
boot window: sample REFRESHES first — a cached fleet made 'settled' meaningless
gates / gates (push) Successful in 9s
Found by live validation on 9201, not by review. GetStacks() is the Manager's in-memory map refreshed by the scheduler every 10s; sampling it every 5s without refreshing means two identical samples can mean the cache did not update rather than that the fleet settled. A container removed ~5s before the window closed was still in the sampled fleet and the sweep logged 'no boot-orphaned apps' for an app that had none. sampleBootFleet now refreshes first; a refresh error degrades rather than aborting the window. |
||
|
admin
|
582135f861 |
v0.190.0 — the boot settle window, both gates on intent, and R-171
gates / gates (push) Successful in 8s
R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix was written). Replacing isBootOrphan's container-count term with recorded intent made a drive-gate-stopped app read as a boot orphan: the gate stops apps with `compose down` (zero containers) and never touches desired_state, because it is not the customer. Observed on 9201 with the drive held unmounted — the sweep found and started it, burned both attempts, and handed it to the dead-app alarm. The write hazard did not materialise (the unbound mountpoint is host-root-owned and the guest is unprivileged) but that protection is accidental and untested. New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive already refuses this; the sweep bypassed it. R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a fleet docker was still restoring — three of six hard resets. Now a settle-then- sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at the end; ends on settled or a 50s budget, and the log says which. The budget is 50s because settle+budget+one retry must stay under the 90s dead-app grace — a test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn rather than the grace being widened to hide it. Widening the window made two more holders reachable, so the one gate covers all three: an absent drive, a quiesce, and an in-flight app-data operation — reusing quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks(). R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way table; absent keeps the old hasContainers behaviour exactly. Its comment argued for the container count and was rewritten. presentStable is untouched. The two gates' agreement is pinned from both sides against one fixture table. 27/27 packages green; 6 red-proofs observed FAIL then restored. |
||
|
admin
|
dbcb306fcf |
v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b)
gates / gates (push) Successful in 8s
The box stops inferring the customer's intent from a container count and reads
what they actually asked for.
Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).
ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.
Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).
Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.
No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
|
||
|
admin
|
4ed938cce4 |
D5: an app restore works from the drive alone (v0.188.0)
The recovery unit on the customer's drive now carries the PORTABLE secret class, so Tier-1/Tier-2 restore no longer depends on the whole-guest tier. A customer needs the drive and nothing else. Part 0's rulings overturned the brief's recommendation, on evidence: - the data_key flag is untrustworthy (4+ encryption keys the catalog itself labels as such are unflagged) -> R-127 - a DB password is not resettable in practice: POSTGRES_PASSWORD is ignored once PGDATA is non-empty, so a regenerated value leaves the app unable to authenticate against its own restored rows while the dump replay still reports success (proven on a throwaway postgres:16-alpine) Ruling (operator): type:secret travels, type:password never does, minus the nonPortableSecrets code register. Plaintext -- withholding the internet- reachable class is what licenses that, and the two are coupled. Precedence: the UNIT WINS over the guest -- the unit's secrets were captured in the same run as the dumps beside them, so they match the data being restored. The fail-closed data-key gate is unchanged. Secret values are never logged; the manifest records NAMES only. |
||
|
admin
|
fd50a73e65 |
C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0)
Both are the system reporting healthy while the customer is not, and both live in the same status-derivation code. Neither is fixed by making the system quieter. C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén." — at the moment the customer pressed it because files were missing, with 156 MB of BookStack's data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit). Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable by any customer action. C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no longer contradicts the alarm on the same screen. README's claim that faults "still surface as restarting" was a wish with no test — corrected in place; it is the seventh such instance. Six red-proofs observed, including the one that matters most: adding StateRestarting to IsDownState fails the brief-restart test with "every deploy and update would page the operator". go test ./... rc=0, 27 packages, run and read separately from this commit. |
||
|
admin
|
3f048e042b |
R-101 + F-DIAG: the restore dialog names the last SUCCESSFUL copy (v0.182.0)
Tier2LastRun is the attempt clock and was rendered as 'Legutóbbi másolat' in the restore confirm dialog. New LastSuccess + SuccessTracked anchor; tier2Update makes the three rebuild sites safe by construction. F-DIAG: six distinct causes, target-aware redaction. |
||
|
admin
|
fb91c8d766 |
F-OBS: the dead-app check gets a positive observable (v0.180.0)
deadapp-check had no observable at default info level: its per-cycle line goes through Scheduler.dbg(), gated on logging.level==debug, so on a default box it is never PRODUCED (not merely filtered) and cannot reach the always-DEBUG ring. A 30s interval also puts it on the scheduler's quiet path. 'No alarms' was therefore indistinguishable from 'the detector never ran' — which undermines confidence in the F-CRIT-1 fix in the field. A periodic summary, not a line per run: at 30s a per-run line is 2880 lines/day, which is why the original author chose silence. Every 20th scan (~10 min) emits one INFO with the scan count, apps evaluated and apps down. A test pins the cadence so it cannot be widened into uselessness. Also corrects the 'unquiesce guaranteed by defer' comment — fault 10 established the guarantee is the crash marker plus Recover(). |
||
|
admin
|
079265ad8e |
F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0)
F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
1. restartAll returned nothing, so the failure was logged and dropped and no
caller could learn a customer's app had not come back. It now returns the
stacks that failed; both call sites record the outcome.
2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
means the user stopped it'). The quiesce loop stops stacks by the same
compose-down path, so a failed restart is also StateStopped and was
whitelisted into silence. Loop.FailedRestarts() is now the only thing that
lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).
F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.
Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).
Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
|
||
|
admin
|
8f46495426 |
seam sweep: move the TieredBackend witness into production code (no version bump)
A witness in a _test.go file fires on go test and go vet but NOT on go build alone — and a build-only step is exactly how the R-88 Part 2 near-miss would have shipped. Moved beside the type it pins, and added one for AgentVersionReporter. No defect found: quiesceBackend and *Client both satisfy their interfaces today. No version bump, no deploy — compile-time only. |
||
|
admin
|
86ea482fc1 |
controller v0.178.0 — R-88 Part 2: only a positive 'never' fires the valve
MinAgent: 0.105.0. scheduledRunAllowed fired on any nil age; it now requires a licence from valveLicensed, which grants it for AgeStateAbsent and for a LEGACY agent, and refuses it for AgeStateUnknown. An unreadable storage no longer masquerades as a first-ever backup and no longer quiesces apps outside the window. A missing wire field means legacy, not unknown — deliberately. Treating it as unknown would stop the valve firing on un-upgraded boxes and starve genuinely new ones. Degrade logged once; unrecognised future values also map to legacy. Caught in passing: TieredBackend is satisfied by a RUNTIME assertion, so the signature change compiled and vetted clean while quiesceBackend silently stopped satisfying it — which would have degraded every box to the single-tier path with no error. Added a compile-time witness. Also corrects the notifier comment that claimed operator-only came from a missing customerMessages entry; enforcement is hub-side operatorOnlyEvents (hub 0.79.0). |
||
|
admin
|
e9c99566b0 |
R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import), wired by an init-only setter because main.go builds the notifier after the loop. Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and recovery rides recordSuccess's existing bool. Uses NEW operator-only event types; reusing backup_failed would have emailed the customer in Hungarian about a backup they cannot act on, since it has a customerMessages entry and is in live enabled_events. Requires hub >= v0.78.0. R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE, consumed at the same single derivation point. 180s grace, derived from the deploy flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app that genuinely fails to come back still alarms. |
||
 Claude Code
|
de96efc0c5 |
v0.174.0 — R-82 Slice B: one quiesce window, two backup tiers
MinAgent UNCHANGED — degrades gracefully against ANY older agent.
The agent gained per-target tiers in v0.97.0. The controller owns quiescing,
so the multi-tier schedule is reconciled here: every due tier is collected up
front and run inside ONE quiesce window (one stop, N sequential backups, one
resume). Two cycles on the weekly night would mean two app outages for one
night's work.
Dedup rule: local-only -> one quiesce; PBS-only -> one quiesce; BOTH due ->
ONE window with both backups inside; neither -> no quiesce.
- quiesce.TieredBackend + BackupTier + ErrTiersUnsupported (optional extension)
- agentapi: BackupTiers/BackupDueFor/StartBackupFor/BackupStatusFor;
targetQuery("") yields an EMPTY suffix so untargeted hits the pre-R-82 route
byte-for-byte
- Loop.resolveDueTiers = the dedup rule in one place, agent order preserved
- quiesceAndPollTiers + pollTier: app stays quiesced until the LAST tier
snapshots (resuming earlier loses app-consistency on the DR tier). Consequence
stated in the docs: both-due-night downtime = first tier's full backup + last
tier's snapshot, which is why tiers run fast-first.
- Manual 'Mentes most' covers EVERY tier, due-ness ignored.
- Window-gate safety valve now uses the OLDEST due tier, so a stale DR tier
cannot be starved by a fresher local one.
Capability detection: /backup/tiers 404 = pre-R-82 agent (the documented
route-probe mechanism). Not a featureProbes row on purpose — the loop needs the
tier LIST, not a yes/no. Degrade logged exactly once per process.
Tests +11, full suite green. Red-proofs #2 and #3 observed and restored.
|
||
|
admin
|
9056f01fae |
v0.173.0 — R-77: endpoint-drift detection, samba protected-set gate, channel log honesty
Source: felhom.eu/documentation/audits/DIAG-agent-channel-2026-07-26.md bootstrap.DetectEndpointDrift names a controller.yaml vs bootstrap.json local_api.endpoint divergence -- one ERROR carrying BOTH values and BOTH paths, its own event type local_api_endpoint_drift, and its own Hungarian banner shown ABOVE the channel banner because drift is the cause and "agent unreachable" the symptom. It writes NOTHING: reconciling from bootstrap.json would clobber a correct controller.yaml on any half-provisioned or hand-repaired guest, so the authority ruling is deferred to R-78. Fail-safe silent on absent/unparseable/ incomplete bootstrap and on an empty endpoint (ensureLocalAPI's fill-if-missing path is untouched). Fingerprint compared as a BOOLEAN only; token never compared, logged or exposed. EffectiveProtected now gates samba on Enabled && UserSet, mirroring BOTH of reconcileSambaAt's early returns, and the doc comment is corrected in the same change -- it claimed "detection and deployment agree in both directions" while citing only !smb.Enabled, an assertion that went false when !smb.UserSet was added. Not over-suppressed: sharing on WITH a password and a dead container still alarms. Channel log: the debounce placeholder is stateUnconfirmed (rendered "unseeded") instead of "up", so a born-down channel no longer logs "up->down" and orUnseeded stops being dead code. Logging only -- the placeholder is still matched in the re-arm condition, so F2 born-down alerting is byte-for-byte unchanged and all nine pre-existing channelhealth tests pass. Tests 951 -> 959, all green. Red-proofs A (both directions), E and F. MinAgent unchanged; felhom-agent untouched. |
||
|
admin
|
2958946517 |
v0.172.0 — R-75: canonical import root, catalog-derived skeleton, import surfaces
${IMPORT_PATH} = <system namespace root>/userdata/import — ONE drop-zone per box,
on the system drive, injected at BOTH compose-env builders with NO per-drive
fallback (unresolvable leaves it unset so compose fails loudly rather than
quietly building a second, dead drop-zone).
Third BindRoot (RootImport) + Import list in BackupSpec, extended through
ValidateBackupSpec/ClassifyBinds. Load-bearing: a stale `userdata: import/<app>`
entry against the moved bind would be a WHOLE-BLOCK reject, taking the app's
mandatory hdd classification with it.
Exhaustive-root audit: resolveAbs/structuralGuard/ComputeCaptureSet/
ComputeFabBuckets now take importRoot explicitly (an import bind resolved
against hddPath would name a directory on the wrong drive); unresolvable is
refused loudly into Skipped. GetImportRoot added to both provider interfaces.
Catalog-derived skeleton: UserdataSkeleton() -> UserdataSkeletonCarry() +
BuildUserdataSkeleton(), SORTED. The carry-list makes zero-removals true by
construction (`documents` is in no catalog app but on both boxes) and is the
fresh-box floor. The sort is not tidiness: the naive map-order derivation
measured 20 distinct outputs from 20 identical runs, which with fbNeedsRecreate
is a fleet-wide FileBrowser restart loop.
One authoritative compose parser: ParseComposeUserdataMounts now delegates to
ParseComposeClassifiableBinds. Import root excluded from per-app migration.
Surfaces: FileBrowser /srv/beolvasas source; app-page "Hova tegyem a fajlokat?"
with PathEscape deep links (never QueryEscape) and class-driven copy;
data_paths: annotation with the Fork-3 asymmetry; system-owned beolvasas SMB
share refused server-side at handler AND store, button omitted in template.
Caught on the way: the sharing template's row struct was function-local, so
adding {{if .System}} would have 500'd every share row. ShareRow is now
package-level and the render test uses the handler's own type.
Tests 915 -> 949, all green. MinAgent unchanged.
|
||
|
admin
|
c97975c1df |
v0.169.0: disk-health card + degradation notification (Lemezek állapota)
Consumes the agent v0.94.0 smart payload (MinAgent floor unchanged; feature-detect by presence). One pure verdict fn agentapi.DiskVerdictFor shared by the dashboard card and the 6h check. Card via a 60s /disks TTL cache (anti-smartctl-storm); unreachable agent -> Nincs adat, page never blocks. disk-health-check (6h) emits disk_health_degraded on a degradation only vs an in-memory baseline (first run silent, recovery/UNKNOWN never notify, multi-attr -> one event). No global banner (deliberate). Pairs with the hub allowlist bump. Tests: verdict table (>=90 red-proof), notifier emit, check first-run-silent (red-proof), degradation-once, recovery-silent, UNKNOWN-excluded, FAILING-critical, nil-smart card, TTL cache. |
||
|
admin
|
82c67e32e1 |
v0.168.0: customer-configurable backup window (Mentési időablak)
ONE setting (window start W) drives every nightly leg at fixed, never-stored offsets: DB dump at W, tier-2 at W+60m, off-box at W+105m (wrap-safe). Precedence settings > controller.yaml db_dump_schedule > 02:30. - scheduler.UpdateDaily: retime a daily job at runtime (no restart) via a per-job buffered resched chan + a select case in runDailyJob. - new pure package internal/backupwindow (LegTimes/GateWindow/EffectiveWindow). - quiesce disk-tier window gate: scheduled cycles run only inside [W+2h,W+6h) with a safety valve (age>cadence+24h runs regardless); manual TriggerNow never gated. Backend.Due now also returns the backup age (from the agent's own /backup/due). - backup page: Mentési időablak card (time input + derived leg/gate rows); POST /backups/window validates -> saves -> UpdateDaily x3 -> flash. Tests: 5 groups, all red-proofed. Agent/cadence//backup/due untouched. |
||
|
admin
|
c23a0f6d2d |
v0.164.0: deliberately stopped apps no longer alarm (banner + email)
A UI stop (Leallitas -> compose down -> StateStopped) is the user's own
action, not a fault, and must not raise the deadapp banner OR the
app_start_failed event. Filter at the single fix-3 derivation point:
extract scanDeployedAppRunStates's pure core to classifyRunStates and
change the down predicate to IsDownState(st.State) && st.State !=
StateStopped. Suppresses StateStopped from both the banner dead-list and
the notifier Down-set at once.
Rests on two invariants (recorded at the seam, README, CONTEXT):
I1 StopStack = compose down => zero containers => StateStopped
I2 P2 census: all catalog services unless-stopped => faults never rest
at stopped (they surface as exited/degraded).
IsDownState unchanged; out-of-band 'compose stop' (containers remain ->
exited) still alerts. Tests +4 (notify 3->4, main 4->7), both red-proofs
verified. No template/funcmap/notifier/counter/copy change.
|
||
|
admin
|
cb8bf14599 |
v0.162.0 — R-71(a): the apply-bridge settle-gate (kills the F10 day-0 race)
The day-0 race (DIAG-f10): a fresh box boots below the operator floor, the apply-bridge consumes the single-use offsite password, then ~35s later the managed auto-floor update replaces the container mid-install -> the new process finds no installed key -> consume -> 404 -> offsite dead until an operator Re-issue. Recurs on every onboarding whose ISO floor lags the managed floor. Ordering-only fix (consume/install/persist internals + the 404-no-oracle contract + the Consumer UNTOUCHED; R-71(b) rejected-by-design): - New seam offsiteapply.SettleProvider.SettleState() + SettleFunc adapter over the self-updater's own GetFloor()/IsUpdateRunning() (no second floor path). - Bridge.AwaitSettle polls 10s BEFORE the 3-min Reconcile ctx: defers while an update runs or the box is below the known floor; GOes at/above floor on the first poll with zero added latency (B'). Bounds 90s floor sub-bound / 5min overall, both GO+WARN (hub that can't serve a floor can't serve a consume -> no burn risk; R-71c is the belt). ReconcileWhenSettled = gate then reconcile. - main.go: bridge goroutine moved after the updater is built; wired only when an updater exists (nil Settle = reconcile immediately, old behavior). Finding: the floor is in-memory (report-ACK ~5-10s), NOT persisted -> unknown on any restart until the first ACK; the 90s sub-bound is sized to that. Tests (injectable clock, fake SettleState, recorded Consumer): A-E + nil-provider + cancelled-gate. Four red-proofs all observed FAIL then restored: gate removed / updateRunning branch / floor sub-bound / overall bound. Deferral paths ship unit-proven + red-proofed, NOT live-fired -- their precondition is now structurally prevented by the v1.25.0 build gate. Layering: gate prevents, (a) defers, (c) heals. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ |
||
|
admin
|
285dd1032f |
feat(v0.156.0): dead-primary alerting (R-51) + boot desired-state reconciliation (R-52)
R-51: aggregateState's mixed branch returned StateRunning ("partial"), so a stack whose
MAIN container was dead behind live helpers alerted on nothing — immich-server sat Exited
for 18 h, 100 % unreachable, no banner and no app_start_failed (audit F4). New
StateDegraded: a DOWN member whose docker restart policy is always/unless-stopped is a
fault (degraded, a down state); no/on-failure is a finished one-shot and stays benign; an
unreadable policy fails CLOSED. The unhealthy/restarting/paused/unknown exclusions are
byte-identical — folding unhealthy into down is the flapping fix-3 avoided.
R-52: new internal/bootrecon — one bounded start-once sweep at startup (2 attempts, 30 s
apart) for apps an interrupted boot left behind, inside the 90 s boot grace so a success
is silent and a failure still alerts. A zero-container stack is NEVER touched: the UI's
Stop is compose down, so a deliberate stop survives a reboot.
Both features carry a production-path wiring test (the v0.154.0 / v0.91.0 inert-seam
class). The main() assertion is an AST walk, not strings.Contains — the substring version
passed its own red-proof, because a commented-out call still contains the string.
Red-proofs run and restored: mix branch reverted -> "running" on the immich fixture;
boot hook commented out -> wiring test fails; zero-container gate dropped -> the
user-stopped app gets started.
NOTE: controller/cmd/controller/ is matched by .gitignore's `controller` entry, so new
files there need `git add -f` (and ripgrep silently skips main.go without --no-ignore).
|
||
|
admin
|
78ff991f1c |
v0.153.0 — R-47: the DB replay no longer races the app, on BOTH restore paths
Closes R-47. No new agent coupling — MinAgent stays 0.90.0. The replay needs a running DB container, so both restore paths started the WHOLE stack first, giving the application a window to rebuild the very schema objects the dump was about to create. Measured live on 2026-07-19 (H4, DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds before the dump's CREATE INDEX, the replay aborted "already exists" under ON_ERROR_STOP=1, and immich reported schema drift. The data survived only because pg_dump emits COPY before CREATE INDEX. Both paths now open a DB-ONLY window: only the stack's database service(s) come up, the dump is replayed with the app still down, and the full start runs only after the replay exits 0. Fail-closed: a dump with no identifiable DB service refuses BEFORE the first mutation. Every exit from the window still does a best-effort full start, so a failed restore never leaves a box with a database and no application. New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty list — argument-less `up -d` is a full start); RedeployFromEnv split into PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden `up -d` inside the old name is what carried the defect on the local path. 19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed effects, replay-failure bring-up, parser decoys, empty-list refusal); three companion red-proofs run and reverted. 23/23 packages green. Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0. |
||
|
admin
|
b5d78d1e0f |
v0.147.0 — feedback slice 1: pressing a button says something
The systemic complaint, twice in one evening: you press a button and nothing happens. No progress, no ETA, no named result. Three worst offenders, fixed on the two patterns already here (deploy 3-step panel, storage-init status poll). No new framework — that is a ROADMAP item; three targeted cards ship tonight. 4a — a verification restore names its result. The flash said the app had been restored "to a verification folder on the drive"; which folder, on which drive, was invisible, so the customer could not go and look at what they had just asked for. Full path now. The restore page gained a listing of existing verification copies (app, size, date, path) — nothing anywhere showed these, so they piled up and the only way to find them was SSH — each with a double-confirmed delete. That delete is the only one this release adds, so it names a STACK, never a path: the Manager resolves the name inside a backups/offsite-restore root it computed itself and refuses anything landing outside. Red-proofed — neutralise the name guard and stack:"" resolves to the offsite-restore ROOT and takes every copy with it. Refusals are asserted as non-effects. 4b — Megosztás enable shows what it is waiting for. Enabling ran ReconcileSamba synchronously inside the POST handler; on a golden without felhom-samba baked that is compose pulling ~100MB, i.e. minutes of an apparently-hung form post followed by "Beállítás mentve." whether or not anything came up. Detached + polled now, distinguishing "képfájl letöltése" from "indítás" — decided BEFORE the work starts, since afterwards the image is always present. Success is probed, not inferred (compose up -d exits 0 on a crash-loop). The password form starts the same job: with UserSet false reconcile deploys nothing, so on a fresh box that is where the pull actually happens. 4c — "Távoli mentés most" streams real progress. restic was already reporting bytes and percent; the runner seam used CombinedOutput() and discarded them. The manual run now passes --json and scans stdout line-by-line: total bytes, percent, current app. Manual only — the nightly stays silent, pinned by a test that fails if it ever passes --json. The poll now arms unconditionally, closing a race the manual trigger always ran: the redirect rendered before the goroutine wrote LastStatus=running, so the poll never armed and the page sat static during the very run just started. Red-proofed twice. Also closes the golden/controller infra-image drift at the source: infra.Images() derives from the existing pins and --print-infra-images exposes it, so the golden bake can stop carrying its own copy. That copy had already drifted — felhom-samba was never added, so the golden baked 3 of 4, which is why enabling Megosztás pulled at runtime in the first place. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE |
||
|
admin
|
d0c1d77491 |
fix(shares): reserved key leaked into the crossdrive_completed hub event
Found by LIVE VALIDATION, not by a unit test: the first demo tier-2 run pushed 'Masodlagos mentes elkeszult: _shares' — the reserved key reached Hungarian customer/operator copy. Mapped at the SOURCE of the notification (RunSharesTier2's tier2Notify calls) so no future notifier wiring can reintroduce it, plus DisplayStackName at the main.go wiring as idempotent defense in depth. Regression test added with a red-proof. |
||
|
admin
|
900c870212 |
feat(shares): R-7b Parts 4-6 — shares restore, samba liveness, UI truth-up
Part 4 — restore: RestoreSharesScratch + PlaceSharesRestore as SIBLINGS of the
per-app scratch/place pair. Files merged missing-only (never overwriting), each
destination PREFIX-ASSERTED against registered LIVE storage roots; definitions
merged with existing-wins; ReconcileSamba via a seam (backup must not import
stacks); credential restored best-effort into the samba named volume.
New routes POST /backup/shares/{restore,place} + a restore-page entry that renders
'Megosztasok', never the raw reserved key.
Also adds scratchJoin: reconstructing an absolute captured path under a scratch
must strip the volume name rather than rely on filepath.Join.
Part 5 — liveness: EffectiveProtected gains a settings-backed dynamic extra so the
samba CONTAINER (not the stack name — they differ) is watched exactly while sharing
is on. FINDING: the issue -> health 'fail' -> existing health_critical event ->
alert -> Hungarian degradation e-mail path needs NO further change, and introduces
no new event type, so the allowlist gotcha does not apply.
Part 6 — UI: per-tier backup status lines on the Megosztas page (amber only on
deviation). Verified the two warning-prose sites (offbox_capture/tier2_capture)
only ever receive per-app stack names, so no mapping is needed there.
RED-PROOFS RUN AND REVERTED (both fired):
4. prefix-assert removed -> place-guard traversal test FAILS
5. dynamic samba extra removed -> Scenario E enabled-case FAILS
|
||
|
admin
|
2eef9b2e4a |
fix(samba): register /api/sharing/ on the mux, not the web switch
The /api/ subtree is routed on the main mux, so the browse case in the web ServeHTTP switch was shadowed by the apiRouter catch-all and 401'd. Moved to ServeSharingAPI behind RequireAuth+CsrfProtect, matching /api/storage/. Found by live validation. |
||
|
admin
|
3286c7faaf |
v0.143.0 — guest RAM resize UI (R-24) — MinAgent: 0.90.0
The customer sees the guest's current memory + allowed range on the Rendszer settings page and resizes it. The controller proxies + maps the agent's machine code to Hungarian; the agent (felhom-agent v0.90.0) enforces every bound and applies live (no reboot). agentapi: GuestMemory + ResizeMemory; ruled 412 -> *MemoryRefusedError (code+bounds); pre-0.90 agent 404 -> typed *StatusError. Capability: FeatureGuestMemoryResize + featureMinAgent 0.90.0 + a featureProbes row (type-asserts GuestMemory so the shared SupportProber/netAgent are untouched). UI (internal/web/system_memory_handlers.go, settings_system.html): "Szerver memoria (RAM)" card + number input; POST /api/system/memory/resize -> capability gate -> agent -> flash. JS confirm only on shrink. Code->Hungarian map; agent-outdated hides the control; agent-unreachable falls back to the guest's /proc/meminfo. Agent English never shown raw. Tests: agentapi (decode, 404, refusal-code, capability table) + web handler (success/ below_usage_floor/agent_outdated). Gates + go build/vet/test all pass. |
||
|
admin
|
596505ed64 |
v0.142.0: offsite repo continuity — orphaned-repo guard (A) + run-status auto-refresh (C)
- Part A: classify restic cat-config failure (wrong-password=orphaned vs no-repo vs other); ORPHANED state + Hungarian card + offbox_repo_orphaned/reset events (once, not nightly); reset = move-aside (never delete) + init, unclaimed auto / claimed confirm. Red-proofs TestOffbox_OrphanDetection_* + ConfirmedReset. - Part C: GET /backup/offbox/status + poll on backups_remote → flips Fut→Rendben/Hiba without manual reload. |
||
|
admin
|
2dd05670ae |
feat(report): Direction-2 immediate-sync wait channel client (v0.140.0)
report.Waiter holds a hanging GET against hub /api/v1/wait?gen=N (same hub URL+key as the pusher). On a generation change it fires the v0.139.0 report.Trigger — nothing else; the report ACK delivers everything through the unchanged machinery. No overall client timeout (held GET); per-request ctx bounds a dead connection. First-observation records-not-fires; same-gen timeout fires nothing; errors (incl. 404 from a pre-v0.58.0 hub) back off 5s->5min while the 15-min cycle reconciles. Wired beside the trigger under the same hubPusher!=nil && Hub.Enabled gate. Red-proof: disable the baseline branch -> first observation fires (TestWaiter_FirstObservationRecordsNoFire), run-fail-reverted. Copy soften: backups_remote/escrow "néhány másodperc, legfeljebb 15 perc". Pairs with hub v0.58.0. Grounding: felhom.eu/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md |
||
|
admin
|
fe9266f53f |
feat(report): v0.139.0 — immediate out-of-cycle hub report on user actions (Direction 1)
New report.Trigger (buffered-1 chan + worker; quiet 2s, min spacing 15s, trailing-edge coalescing) generalizes the v0.70.0 geo out-of-band push. One canonical fire closure in main.go; wired: geo save/sync + app deploy/remove/delete (api reportPushNow), escrow recovery-code claim, notification-prefs save, app-email toggle, offsite config + per-app toggle, customer claim (web SetReportTrigger seam, nil-safe, fired only after a successful local commit). 15-min hub-report cycle untouched as the reconciliation backbone; hub.enabled=false stays a strict no-op. Tests: trigger engine (2 red-proofs), seam fires-after-commit-only, nil-seam no-ops. |
||
|
admin
|
120332103a |
feat(escrow): v0.138.0 — "awaiting hub confirmation" waiting state
After a completed escrow ceremony the Távoli mentés page showed the yellow "Helyreállítási kód szükséges" card for ~15 min until the next hub-report ACK flipped pending→escrowed. Phase-0 diagnosis (read-only) = verdict A (report-cycle lag), already resolved on the demo box (escrow_state:"escrowed"); hub Hypothesis B verified false (SaveHostEscrow ON CONFLICT already clears stale_at on upload) → no hub change. - settings.OffboxTarget.CeremonyCompletedAt: stamped on the recovery-code claim, zeroed on the auto-confirmer Flip + the deprecated manual confirm; persisted. - web/handlers.go: offboxCeremonyWaitState + escrowCeremonyGraceWindow (35m). - backups_remote.html: info "megerősítésre vár, legfeljebb 15 perc" card → warn "a megerősítés nem érkezett meg" past the window. Existing branches untouched. - backups_escrow.html: "Mi történik ezután?" note on the wizard's final step. - Test web/escrow_wait_state_test.go (truth table + red-proof recorded in REPORT). No scheduler/agent/hub/endpoint changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp |
||
|
admin
|
cf9ce01917 |
.fab exclusion scoping: classes in the manual export (Task 4, v0.136.0)
SQ6 over-capture FIXED for classified apps: the userdata root tar is exclude-scoped (keeps only ancestors/descendants of a SELECTED bind relpath — R1-C, the tier2Reconcile keep-rule); no selected userdata bind → no root tar (radarr state-only). New appbackup.ComputeFabBuckets (shared resolveGuardCollapse pipeline; class buckets; guards over ALL classes; no cross-bucket containment). appexport/fabplan.go: computeFabPlan + tarDirectoryExcluding + fabEstimateSplit. ExportRequest gains DeselectOptional/OptInExcluded (both start handlers — two-call-site); mandatory is a server-side floor. Manifest v1 + import UNTOUCHED; legacy apps byte-identical to v0.130.0. Estimate additive class split; export UI: locked-mandatory/optional-checkboxes/excluded-opt-in + two-number warning. All 6 §10 red-proofs verified. 6D Accept #1 now runs against this shape. |
||
|
admin
|
2d20859858 |
Offsite tier policy engine: mandatory userdata, raw-data quota, restore rework (Task 3a, v0.134.0)
Each toggled app's offsite push = one multi-path restic snapshot (recovery unit + TierOffsite mandatory userdata via ComputeCaptureSet); legacy/undeployed stay unit-only. Loud capture gaps (SP-3.4: restic 0.14.0 silently skips missing paths). Quota = stats --mode raw-data (SP-1; displayed size drops once). Pre-push enlargement gate blocks the userdata enlargement over-quota (unit-only push continues; EnlargedBlocked; edge-triggered notify). forget --group-by host,tags on both sites (SP-2). Restore reworked: scratch off the rootfs + headroom gate (F-A1), unit-only default via --include, size-first full, place-to-live missing-only merge (never --delete). UI: unit/full-two-step/place actions + per-app blocked note; route POST /backup/offbox/place. HUB FLAG: offbox_enlarge_blocked event needs hub allowlist for push delivery. +13 tests; all 10 §10 red-proofs verified. No tier-2/.fab/hub/agent changes. |
||
|
admin
|
0649f9a3e6 |
Backup classification: schema + parser + pure classifier (INERT, v0.132.0)
Task 2 of the backup-classification-redesign arc. Ships the referential-
coupling classification as DATA + PARSER + PURE CLASSIFIER, deliberately
inert — no backup tier changes behavior. Task 3 (tier policy engine) and
Task 4 (manual .fab UI) consume it.
- appbackup/classify.go: BackupSpec/BindSpec/ComposeBind/ClassifiedBind;
ClassifyBinds (SQ5 two-level default — explicit beats :ro; unlisted
writable→mandatory, unlisted :ro→excluded; nil spec→legacy/false);
ValidateBackupSpec (whole-block-reject on any defect, first defect named).
- stacks/classify_binds.go: ParseComposeClassifiableBinds — ${VAR}-relative
binds + :ro flag (NOT ParseComposeHDDMounts/ExportDataMounts, the traps).
- Metadata.Backup + LoadMetadata as the single validation choke point (bad
catalog block → nil + one ERROR → legacy, within one sync cycle).
- Manager.ClassifiedBinds + StackDataProvider.GetStackClassifiedBinds seam
(delegated by stackAdapter, nil-stubbed in every fake) — wired + tested
now so Task 3 consumes a tested seam.
INERT: full pre-existing suite green with zero test-logic edits. +14 tests;
red-proofs RP-1..RP-4 confirmed. The 13 catalog backup: blocks ship in the
same app-catalog change (this controller deploys first).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
|
||
|
admin
|
68f0e0cf5c |
F-S2 + F-S3: compose-derived appdata dir resolution (v0.131.0)
The controller assumed an app's HDD appdata dir is always appdata/<stackName>.
paperless-ngx writes appdata/paperless (stack paperless-ngx), so every consumer
keying by stack name silently missed it via a stat-and-skip. One canonical
resolver appbackup.AppDataDirNames derives the real dir name(s) from the app's
compose ${HDD_PATH} binds; all consumers use it.
- F-S2 (tier-2): RunTier2 mirrors the resolved appdata/<name> (paperless docs
got NO tier-2 copy before). Tier2Info size + RestoreTier2Files live dir use it.
WARN when a declared appdata dir is absent. New tier2Mirror seam.
- F-S3 (migrate, NEW): all six per-app appdata legs (collision/size/copy/verify/
cleanup/skip-set) now loop resolved names. scope="app" migration of paperless
previously copied nothing and left an empty media dir (scope="all" was saved by
the merge walk). WARN on missing declared dir in the copy leg.
- Multi-dir (N>1) refusal: tier-2 backup/info/restore refuse loudly (Hungarian);
migrate supports N. No catalog app hits it today; lifted by Task 3.
- Display: storage page sums resolved dirs.
- Truth repair: the v0.130.0 "tier-2 copies the namespace wholesale" claim is
false; corrected in CHANGELOG + main.go export-adapter comment.
+9 tests; red-proofs RP-1..RP-5 all confirmed. Controller-only, no agent/hub
coupling. Task 1 of the backup-classification-redesign arc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
|
||
|
admin
|
8967ba7cb4 |
C6B-F1 cause 2: .fab export mount discovery resolves the userdata convention
exportAdapter.GetStackHDDMounts now returns stacks.ExportDataMounts — the
${HDD_PATH} binds UNIONed with the ${USERDATA_PATH} ROOT (captured at the
root, not per-bind, so the manifest's basename keying round-trips through the
existing import mapping without touching restore). Containment-aware dedupe
both directions. The backup-side stackAdapter is intentionally unchanged.
Red-proof: pre-fix behavior fails TestExportDataMounts_UserdataConvention/
MixedBindsUnion/LiteralUserdataBindDeduped (run->fail->revert recorded).
|
||
|
admin
|
db16371f47 |
v0.128.0: chunked browser .fab upload on /import (tunnel-proof)
Cloudflare edge caps request bodies (~100 MB, probed live: 120 MiB -> edge 413,
80 MiB -> origin), so the client slices the file into 64 MiB strictly-sequential
chunks; the server streams each to a .part file in the default drive's exports
dir and finalize renames atomically. Scan/validate/import pipeline untouched.
upload/{init,chunk,finalize,abort} inside ServeExportAPI (inherits auth+CSRF);
single-flight; offset==received or 409+echo; free-space gate; collision ->
lowest-free "name (N).fab"; startup GC of *.part-*; 15-min idle abort.
appexport.DiskFree exported (seam web.uploadDiskFree). Scenarios A-F tested,
red-proofs run (traversal / out-of-order / overwrite).
|