main
464 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 admin
|
c6b69d888e |
v0.205.0 — a run that skipped an app the customer selected is not successful (R-234)
gates / gates (push) Successful in 21s
THE VERDICT. The R-203 block already said "a warning beside a success is read as a success" and applied it to ONE of the two shapes it describes: an app missing a declared mandatory FOLDER made the run incomplete, while an app skipped ENTIRELY still reported ok. Both do now. Which skips count, decided by measurement: selected+deployed with no recovery unit YES; selected but NOT deployed no (named, with what to do — a box left amber by an app somebody removed is a status nobody reads); disconnected/decommissioned drive no (own signal); nothing selected no. LastSuccess and SnapshotCount still record what WAS captured. THE FILED MECHANISM WAS NOT THE MEASURED CAUSE, and saying so is the point. §3 stated that toggling an app on leaves it without a bundle so the first run skips it. Measured on demo-hp: the run's own pre-dump phase calls captureAllRecoveryUnits for every DEPLOYED stack, through admitApp, before the push — a unit moved aside was RECREATED and the run reported ok. That state does not survive a run. What actually produced the 2026-08-06 sequence: the manual run was dropped by the single-flight while an earlier run was still going. runOffboxBackup returned nil, the handler had already answered "A tavoli mentes elindult", and the card then showed the PREVIOUS run's green verdict — read as covering the app just selected. The decision is now taken synchronously in the handler and a dropped request says so. The nightly path still returns nil on purpose: nobody asked, and it retries. §7.3 measured before deciding: CaptureRecoveryUnit writes a few KB of compose + manifest, only ENUMERATES dumps rather than creating them, is idempotent and does NOT stop the app — and already runs inside the off-site run. So there is no wait to remove for a deployed app and NOTHING was built. 28 packages ok, 9/9 gates. Four red-proofs, each asserted to have applied. Fixture note: the shared provider's ListDeployedStacks returned nil, so Scenario A first passed for the wrong reason; fixed with an opt-in deployed set that defaults to nil. |
||
|
admin
|
53e9bf0224 |
v0.204.0 — the restore list is keyed on the store (R-237); the size gate stops refusing in silence (R-238)
gates / gates (push) Successful in 26s
R-237: /backups/restore listed apps that are CURRENTLY DEPLOYED and CURRENTLY TOGGLED ON for future off-site backups. A rebuilt box has neither, so a household that had just lost everything was shown nothing to restore while the repository held their snapshots — measured live on the R-201 re-walk. To restore an app you had to select it, to select it you had to have installed it, and to know what to install you had to see the backup you could not see. The store is now the source of the list (offsite_restore_list.go), built on the existing R-193 OffsiteInventoryList. Installed-ness became a property OF a row, never a filter on it. Every case is answered rather than hidden: a snapshot for an app that is not installed is offered and says it will reinstall first; an installed app with no snapshot is shown as having nothing; an unreadable store renders as UNKNOWN (R-225's rule, one screen over) AND keeps the action, because "we could not look" is not "there is nothing"; no-target is its own state. The felhom-offbox and _shares marker tags are excluded from the app list. R-238 classified as a HARNESS ARTIFACT: mode=full without confirm=1 is step 1 of a deliberate two-step — it starts no job by design and redirects carrying &full_prep=<app>, which deriveWizardStep requires to reveal the commit. A driver that did not carry it forward landed back on the intent step. The operator's browser run completed the same restore. The wizard's precedence rules were NOT re-keyed: a stale ?full_prep= must never resurrect a commit button mid-restore. The residue WAS real and is fixed: neither branch of that step wrote anything to the log, so a refusal — including by the headroom gate — left no trace on the box. Both branches now log, and so does the concurrent-op refusal. resolveWizardApp is removed: it was dead once the gate moved, and its test pinned the defect's behaviour (an untoggled app refused), which would have read as policy. 28 packages ok, 9/9 gates OK. Three red-proofs, each asserted to have applied. |
||
|
admin
|
9dc26459ea |
v0.203.0: the box collects what the hub staged for it (R-218 consume half) + R-220's message
gates / gates (push) Successful in 10s
R-218's declaration half shipped in v0.201.0 and works. Its consume half never existed. Reconcile ran exactly twice per process — at start-up and when the recovery screen drives it — and BOTH fire before the hub has anything staged, because the hub stages in RESPONSE to the declaration those runs precede. Measured on the R-201 re-walk: unlock reconcile 11:43:07, hub staged 11:44:57 saying 'next cycle', a full report cycle ran 11:55:46, still unconsumed at 12:06. A guest command line applied it in 18 seconds — everything correct except the trigger. Bridge.RetryIfDeclared re-runs the SAME reconcile on a 5-minute tick, driven from the box's own published declaration (OffboxReportStatus().State) — the very statement the hub acts on, so the two cannot disagree. Poll, not an ACK flag, decided on the promise: the no-target message says 'amint megvannak' (no deadline) and the card says 'within a day'. Five minutes is inside both by a wide margin and needs no hub change. It stops by construction — a healthy box does no work and logs nothing — and the settle gate is deliberately kept via ReconcileWhenSettled. The marker was investigated and left alone: applied_marker lives in the guest's DataDir, which a rebuild destroys, so it cannot suppress a legitimate re-run. R-220's customer half: the refusal no longer tells the customer to choose from a list that may be empty. It names the rebuild, points at the Meghajtók page, and promises no outcome. Red-proofs: remove the retry -> credential uncollected (the dead end reproduced); drop the stop condition -> a healthy box hammers the hub; call Reconcile instead of ReconcileWhenSettled -> settle gate bypassed; restore the old sentence -> the impossible action returns. 28 packages ok, vet clean, all controller gates OK. |
||
|
admin
|
c7446f2d6a |
R-225/R-227/R-228 Parts 2-4: unknown is not zero, the gateway speaks Hungarian, the set-aside is visible
R-225 — an unread store said '0 pillanatkép / 0 / 50 GB' above a card stating it held backups under another key. An SFTP listing found snapshot f3d9cd67 and 12 535 KB really there; snapshot_count and repo_size_bytes were simply ABSENT and the zero value spoke for them. StatsKnown is now NAMED, for the same reason OffsiteInventory.Empty is: zero is what an unread store and an empty one both look like, and on the wire 'absent' and '0' are the same bytes. The fill bar renders only when the fill is known — a 0%-wide bar is a picture of emptiness, and a picture is a claim. A measured zero still says zero. R-227 — WHICH LAYER ANSWERS: traefik, and this repo generates its config. But traefik v3 serves no static files, so a branded proxy page needs a new always-up container for every 502 on the box — out of proportion, and scoped in the report rather than built. Shipped instead: the unlock posts via fetch and answers a gateway failure in Hungarian without leaving the page. Progressive enhancement — with no JS the plain POST is unchanged and still shows the proxy's error, which the report says plainly rather than implying otherwise. R-228 — the set-aside history was recorded in orphaned_renamed_to and read by nobody: a census found zero references in any template or handler, while 12 535 KB sat at that path. It is surfaced as two facts and stops. It does NOT promise the history can be reopened, because it cannot be by anyone today (R-199's inventory is unbuilt) — and the set-aside CONFIRMATION copy was corrected for the same reason: 'a helyreállítási kód nélkül többé nem lesznek megnyithatók' implied that WITH the code they could be. The field's own comment called it 'recovery-code-recoverable', which was the same over-promise in the code. Tests: scenarios F, G, H as render tests per branch of each gate. Red-proofs, each demonstrated failing then restored: remove the StatsKnown guards (F, 'R-225 RETURNED: an unread store reports a snapshot COUNT of zero'), delete the set-aside block (H). The F assertion on the fill bar is scoped to the bar's own container — a bare width:0% search matched unrelated elements and would have passed for the wrong reason. 28 packages ok, vet clean, all controller gates OK (the emoji gate caught a warning sign in a template comment). |
||
|
admin
|
1e759a16ec |
R-224/R-226 Part 1: why the unlock failed decides what we say
The failure branch was a two-way choice — superseded? M4 : M1 — and BOTH are statements about the customer's code. rerr was never inspected, so a hub that refused, an agent that was stopped and a genuinely mistyped code all produced the same accusation. Measured live 2026-08-05 with a CORRECT current code: hub firewalled off 0.0556s, agent stopped 0.0299s, against ~1.0s for a real unseal. Five classes, from the VALUE and never the text: hub-unreachable 502/503 from the agent — the code was NOT used agent-unreachable no agent verdict at all (transport) — NOT used no-bundle 404 bundle-too-old 409 asked-and-refused 400 — the ONLY class that may mention typing unknown everything else -> NEUTRAL, the safe default agentapi.RecoveryRefusal carries the status as a value (refusalError flattened it into a sentence, and a sentence is not something a caller can branch on). THE OLD-AGENT CASE IS WHY THIS NEEDS A COUPLING. Agent < 0.126.0 answers 400 for both a fetch failure and a wrong code, so a 400 from one cannot be read as a refusal. FeatureRecoveryFailureClass (MinAgent 0.126.0) withholds that reading and the 400 degrades to neutral. The gate BLOCKS NOTHING — it only decides whether the customer may be told to check their typing. R-226: the superseded message now names BOTH possibilities and restores the ten-words prompt. The two are indistinguishable at the engine; the honest message says so. It still does not promise the earlier package can be opened. Elapsed time is logged (it is what diagnosed this) and is NEVER a classifier. Tests: scenarios A-E at the HANDLER + the classifier table asserting the same sentence under two statuses classifies two ways. Red-proofs, each demonstrated failing then restored: delete the 502 case (A), remove the mistype clause (C), default to the accusation (D), route an instant transport failure to the typing message (E). Two existing tests encoded the defect and were corrected, not deleted: the web fake returned a BARE error for 'wrong code' (which is the shape of a failure we cannot classify), and R-222's test forbade any mention of typing on a superseded box — half of which R-226 deliberately reverses. 28 packages ok, vet clean, all controller gates OK. |
||
|
admin
|
a3499d1807 |
v0.201.0 — a correct recovery code is never called wrong again (CAMPAIGN-11) — MinAgent 0.125.0
gates / gates (push) Successful in 9s
R-216: the offsite key recovery is a coupled feature and now says so. featureProbes +
featureMinAgent 0.125.0 + a Supports gate at the unlock entry point, FAILING CLOSED — an
agent that cannot answer is named as such instead of the customer's code being blamed.
Measured live: a 404 from agent 0.120.0 came back as "we did not accept your recovery
code, check that all ten words", in 0.134 s, against a perfect code.
R-218: delete the repo-password short-circuit in needsOffsiteCredential. The declaration
stops when the TIER WORKS, not when a key exists — installing a key is the recovery
screen's whole job, so succeeding at recovery was switching off the mechanism that would
have delivered the coordinates to use it.
R-219: the unlock finishes the job — place the key, bring the tier up, then list. Without
it the promised listing could never render on the shape the screen exists for.
R-217: an unreadable store no longer claims to have opened with unattributable content
(the OffsiteInventory{} zero value). Opened / empty / unreadable are three states.
R-222: a code that is right about a RETAINED earlier package is named, not blamed. States
what the hub knows and promises nothing — no read path exists.
R-215: GET /recovery is gated on the same predicate as the interception.
Five red-proofs, each demonstrated failing and restored.
|
||
|
admin
|
636c51e542 |
R-193: the recovery screen — unlocking, and only unlocking (v0.200.0)
A customer whose machine was rebuilt had everything needed to get their data back and no way to find out: the only route was a command line. This is the screen that closes that. IT UNLOCKS, AND ONLY UNLOCKS (operator ruling). It explains, takes the recovery code, opens the repository and shows what is in there — apps, dates, sizes. It restores nothing: restore is already per-app and lives in the backups area, and a screen that unlocks and then offers to overwrite is two decisions wearing one button. ONE CORE, TWO CALLERS. RecoverInstallCore is split out of RecoverAndInstall; the CLI wrapper keeps its exit codes and printed lines byte-identical, and the handler drives the same function. Two implementations of the one operation that can permanently lose a customer's data would drift, and only one would be tested. Asserted from source on both sides by AST. THREE WAYS OUT, none a dismiss button: recover; 'most nem' (the full page stops interrupting, the backups-area entry point stays PERMANENTLY, bound to the offer and never to the postpone flag); and 'I do not want the old data' — confirmed TWICE and reaching the SHIPPED move-aside, which sets aside and never deletes. THE CODE IS HANDLED NO MORE LOOSELY THAN ON THE COMMAND LINE: POST body only, never logged, never persisted, never echoed, cleared on every path, no-store, autocomplete off. No lockout — the code is a ten-word phrase, and locking a customer out of their own data for a typo is worse than anything it prevents. TWO DEFECTS THE TESTS CAUGHT, both fixed: an UNCLAIMED (legacy-open) box would have been shown the page, because RequireAuth passes such a box through; and the inventory nil-dereferenced when no off-site target was configured, which is exactly the pristine rebuilt shape. |
||
|
admin
|
1214bae0a2 |
R-204 item 4 (box half): a rebuilt box DECLARES that it needs a credential (v0.199.0)
An absent off-site object has four meanings — never configured, mid-restart, a transient config read failure, and rebuilt-and-stranded — and the hub cannot tell them apart. The box can, from two local facts it holds with certainty, so it says so instead of leaving the hub to deduce it from a silence (operator ruling). The ACK's identity_blob_present is now recorded on EVERY ACK, before the gates that used to discard it: on a box with no off-site target the auto-confirm returns immediately, which is exactly a rebuilt box, so the one fact distinguishing it from a box that never had off-site backups was thrown away every cycle. The declaration needs BOTH halves — a fresh data area AND a hub-held recovery package. Freshness alone is a box that never had off-site backups; dropping that condition makes the whole fleet ask for credentials, which is what the Scenario B test exists to catch. The object carries enabled:false and zero sizes, which is what makes it inert to the hub's existing fill and staleness checkers and to a pre-upgrade hub. A configured box's JSON is byte-identical to v0.198.0's. |
||
|
admin
|
2e936f43bf |
R-204 item 3: a restore says what it restored, and what it did not (v0.198.0)
mode=unit restores the recovery unit — the app's definition, configuration and database dumps — and NOT the customer's own files: RestoreOffboxScratch passes --include <unit path> and the userdata in the same snapshot is excluded by it. The outcome was one sentence for both modes and named neither scope, so on the last step of a disaster recovery the customer was told the app had been restored after the thing they were looking for had not been. restoreScratchOutcomeMsg states what came back, what did not, and the next step that gets it. The wizard's intent card states its scope before the choice. The full-restore size gate is untouched and pinned as unchanged; the default stays unit, since all three wizard forms set mode explicitly. |
||
|
admin
|
73b6dbc27d |
R-204 item 1: a freshly minted reset code works without a restart (v0.198.0)
--print-reset-code runs as a separate process and persists the new code; the running server's cache was never told, so the code the customer was told to type was refused until the controller restarted. Nothing said so — during the 2026-08-04 drill that cost two attempts with an operator present. effectiveClaimCode now reads through to the persisted state before applying the settings-vs-config precedence, which is itself unchanged. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works, which is worse than the bug. Fails closed on an unreadable state; an absent file is not an error. |
||
|
admin
|
58c703bd44 |
R-203 Part 2: a run that missed a MANDATORY directory is not a successful run (v0.197.0)
gates / gates (push) Successful in 8s
The gap was already detected and warned about, in Hungarian, naming the app and the folders -- that warning is what stopped the R-201 drill. The defect was that the run still reported `ok` beside it, and a warning standing beside a success is read as a success. last_status gains "incomplete": minted, because "ok" | "error" | "running" had nothing meaning "it ran, and this app is not fully protected". NOT "error" -- the rest of the run worked and what was captured is real, so SnapshotCount and the LastSuccess anchor still record it. Half a backup is not no backup. The gaps are now recorded STRUCTURALLY (offboxRunResult.mandatoryGaps), not only as prose, so the verdict has something to act on. It reaches the operator through the EXISTING per-run digest (backup_run_failures) rather than a new event type -- a new type is a two-repo change and the hub drops anything outside allowedEventTypes. The stat-filter gains the ClassMandatory check Tier 2 already had. It is a NO-OP today (TierOffsite admits mandatory only), so no customer-visible warning disappears -- demonstrated by widening the tier filter alone and watching the check hold the line. ANTICIPATED: calibre-web on demo-hp has exactly this gap, so its off-site status becomes incomplete the moment this ships. That is correct and is the point. Red-proofs: my first Scenario-C proof PASSED because the test only reached offboxCaptureSet while the mutation lives in runOffboxInternal -- a mutation the test cannot observe is not a red-proof, and the fix was the test. The run-level test now fails under both mutations (unreachable gap recording; unconditional ok). |
||
|
admin
|
a96c3d9473 |
R-203: the export-mount resolver takes the namespace root too (its own commit)
gates / gates (push) Successful in 9s
ExportDataMounts lives in delete.go, which reads as a destructive path. IT IS NOT: its single production caller is the .fab export adapter, and nothing deletes based on its result. The delete path's own guard, ProtectedHDDPaths, is layout-agnostic by construction -- it protects BOTH <hdd>/... and <hdd>/felhom-data/... -- so deletion was never affected by the namespace-root defect. That scope note is now in the function's doc comment, because the file placement will mislead the next reader exactly as it misled the spec for this change. Separated into its own commit anyway, so a change to a function whose filename says "delete" is reviewable on its own. An empty nsRoot falls back to hddPath -- the pre-R-203 shape -- so any caller not yet updated keeps working on enrolled drives. Tests cover both drive kinds and assert the NEGATIVE: no emitted path lies outside the app's own data roots. Red-proof: leaving the site bare fails the system-drive row, emitting /mnt/sys_drive/userdata where the canonical root is /mnt/sys_drive/felhom-data/userdata. |
||
|
admin
|
73efb091d9 |
R-203: the app and its backup look in the same directory — one resolver, every caller
gates / gates (push) Successful in 9s
appbackup's path helpers take a NAMESPACE ROOT. Five call sites passed a bare DRIVE path.
On an enrolled drive the two coincide, so nothing showed; on the system-data fallback they
differ by exactly the felhom-data segment, and the app then bound a directory the off-site
capture set never looked at -- while the run reported ok. Measured live on demo-hp: the app
wrote to /mnt/sys_drive/userdata/media/books, the capture set looked for
/mnt/sys_drive/felhom-data/userdata/media/books.
THE RULE NOW HAS ONE EXPRESSION. appbackup.NamespaceRootFor / IsEnrolledDrive encode the
drive-kind comparison; backup.Manager.namespaceRoot and stacks.Manager.inGuest delegate to
it. There were already TWO copies and they differed -- the backup package's compared without
filepath.Clean, the stacks package's with it, so a trailing slash from config would have
flipped the mode in one and not the other.
Sites routed through it:
- stacks/deploy.go withPathVars -> ${USERDATA_PATH} (the live defect)
- appexport/fabplan.go + export.go (via a new provider method)
- web/handlers.go FileBrowser mounts (latent: the system drive is
deliberately never a registered StoragePath, so this is the identity today)
ComputeFabBuckets now receives the namespace root, which is what ComputeCaptureSet has always
received -- so the export's classified paths and the backup's capture set describe the same
directories by construction instead of by coincidence.
Tests are table-driven over BOTH drive kinds, because this survived by being invisible on the
kind that already worked. Red-proofs observed: restoring the bare-path call fails the
system-drive row with the two paths differing by /felhom-data; inverting the drive-kind
comparison fails every enrolled row.
|
||
|
admin
|
1b1366bb6e |
controller v0.196.0: the recovered key installs itself (R-200 plumbing half) -- MinAgent 0.125.0
gates / gates (push) Successful in 8s
--recover-offsite-install is the sibling of --recover-offsite-check: same fetch/unseal path through the agent, same STDIN discipline for R, but it PLACES the recovered repository password via InjectOffboxPassword so a rebuilt box reopens the history it inherited. Doing this by hand would put the offsite DATA key through a terminal, a clipboard and shell history. In-process the value goes agent -> this process -> the 0600 file and is rendered nowhere. The confirmation is a SECOND invocation: without --confirm-install it prints both hashes and writes nothing, so the operator sees the comparison before any write is possible. Three outcomes, named distinctly: installed (no local password -- the rebuilt-box shape), unchanged (identical key already present, nothing written), refused (a DIFFERENT key present; installing would clobber the key the current repository is encrypted under, and no force option is offered). Exit 2 for the refusal, distinct from 1 for a failed step. Red-proof: removing the confirmation gate makes the dry run write, failing the test. The R-persistence test carries a positive control -- a planted copy is found, then removed and not found -- because an absence check is worth only what its sensitivity is. |
||
|
admin
|
9640e51321 |
controller v0.195.0: prove the offsite key comes back (R-200 plumbing half) -- MinAgent 0.125.0
gates / gates (push) Successful in 10s
--recover-offsite-check is a docker exec diagnostic in the shape of --print-reset-code: it reads the customer's recovery code from STDIN, asks the agent to fetch this host's sealed bundle and open it, and reports whether the recovered key matches the one on disk BY SHA256. Two hashes and a verdict; never a password, never R, never a blob. R comes from stdin and not a flag because a flag value is visible in ps, in shell history, in a container's command line and in any transcript of the session that ran it. IT COMPARES; IT DOES NOT INSTALL. The recovered password is never written to offbox/repo_password -- installing changes a live box on a path nobody has walked, and that link is next session's, with the drill around it. A test asserts the data dir is byte-unchanged after a check; its red-proof (adding the install call) fails it. Exit codes: 0 match, 2 clean MISMATCH, 1 a step failed -- "it failed" and "it worked and disagreed" must never share a status. A box with no local password reports distinctly: that is the rebuilt-box shape, where the next step is to install rather than compare. Nothing customer-reachable ships here: no card, no form, no preview. |
||
|
admin
|
88897a224e |
v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)
gates / gates (push) Successful in 8s
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed events reached the hub and TWO operator emails went out. The hub's operator cooldown key is customerID:eventType(+tier) and that event carries `app` but no `tier`, so the key held no app identifier: the first refused app took the hour's slot and every other app's failure was discarded BEFORE anything was written down, leaving no row on any channel. The obvious fix — put `app` in the key — was ruled against: on a full disk it produces one email per app, the volume problem wearing the correctness problem's clothes. internal/backup/runsummary.go: a per-run collector with exactly admissionSet's lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the end and only when something failed. A clean run emits nothing. The per-app event stays and becomes the RECORD — the hub routes it record-only, stored and logged every time, never competing for an email slot. The record and the notification are now different things. Deliberate skips (disconnected, decommissioned) are excluded: they have their own alert, and a nightly email about an unplugged drive is one the operator learns to ignore. A manual run always reports: the digest carries a unique run_id the cooldown cannot collapse. Someone pressing the button is actively trying to get a backup. THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a capture failure found between runs would be recorded and never notified — a new silence introduced while closing one. That path emits a digest with NO run_id, so the ordinary 1-hour cooldown caps it exactly as before while the mail now lists every failing app instead of whichever was first. A refusal is recorded ONCE, where the verdict is taken, not at the three legs that consult it — R-181's contract is one verdict per app per run. Noting it per leg listed one refused app three times and produced "2 of 1 apps failed". Found by the digest's own test, not in review. Silence is safe because the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail this box sends (monitor/deadline.go:396,417). Confirmed, not assumed. 7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first attempt — the AST test walked the backup package and not main.go; the test was fixed and the mutation re-run rather than the pass recorded. |
||
|
admin
|
6c43bf6156 |
v0.193.1 — the refusal's size estimate is rendered in bytes, not "0.00 GiB" (R-181 follow-on)
gates / gates (push) Successful in 9s
Found by v0.193.0's own live proof run. The estimate was printed fixed to two decimal GiB, so every app under ~10 MB rendered as "estimated 0.00 GiB write" — which reads as "no estimate was available" and is the opposite of what happened. Observed live on demo-hp 08:59:46: opengist's real 178 KB estimate printed as 0.00 GiB. Shipped in the same session because it is the same defect class R-181 is about: a message an operator cannot rely on is worse than no message. The arithmetic is unchanged and still in GiB — the reserve's own unit, so the comparison against FloorFreeGiB reads directly. Only the rendering moved to humanizeBytes. estimatedWriteGiB -> estimatedWriteBytes, with the GiB conversion done once at the point of comparison. |
||
|
admin
|
fef07c3923 |
v0.193.0 — the reserve guards the write that fills the disk, and its promise is true (R-181)
gates / gates (push) Successful in 9s
B2's capture floor (v0.192.0) was consulted in exactly ONE place — captureAllRecoveryUnits, which writes a few KB. The two legs that write the BULK into the same backups/primary/<app> tree, the DB dump and the volume dump, ran FIRST and unguarded. Measured live on demo-hp 2026-08-03 06:40:03: opengist's volume dump wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then refused the cheap write it had already lost the argument to. Its refusal message claimed "the previous unit is untouched" — measured false: that app's tar had gone 182,272 B -> 2,147,666,432 B under a stale manifest. Sixth entry in CLAUDE.md's table of shipped guarantees the code did not provide. Fix: ONE admission verdict per app per run (internal/backup/admission.go), taken before that app's FIRST write and covering all three legs — they write under one per-app root, which is why one verdict can honestly cover them. - Lazy, at the app's first write, NOT once at run start: app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. - Remembered for the run, never re-decided between an app's own legs — that is the split this closes. Reset per run. - Placed ahead of DumpAppVolumesSafe, which stops the stack as its first act, so a refused app is never bounced. After the volume-less check, which has no write. - Exactly one operator alert per refused app per run. - Leg order unchanged: volume dumps still precede the capture. The floor is now SIZE-AWARE: it asks whether THIS app's write would cross the reserve, not only whether the filesystem is already below it — which is how an app was admitted at 96% and then allowed to write 2 GB. Estimate = the app's previous .sql + .tar on disk. No history -> headroom-only, deliberately, and the alert says so. A container-based du per volume was MEASURED and rejected: 66 timed runs on demo-hp guest 9201, median ~355 ms/volume (341-404) on volumes holding tens of KB — container start-up, not the walk. Decisive on top: docker run needs the writable layer, so it can fail under exactly the pressure the reserve handles. The message was NOT weakened; the behaviour was moved so the wording became true. It now also names which term bound. Every claim is checked against a sha256 fingerprint of the tree it describes, never against the log line. Still refuses and never deletes: nothing here is generational. 11 new tests through the production functions. The DB leg cannot run without Docker, so its gate is pinned by an AST walk of backup.go asserting admitApp precedes DumpOne (strings.Contains is insufficient — a commented-out call still contains the string). 4 red-proofs demonstrated failing then restored. |
||
|
admin
|
4be6467b50 |
v0.192.0 — the capture floor replaces the bulkhead (R-165, decision B2)
gates / gates (push) Successful in 8s
Ships BEFORE the disk-layout merge it exists for, and is harmless on a box that never gets it. The mp1 partition was a BULKHEAD as well as a ceiling: it kept a runaway capture from filling the space the container runtime needs, because /var/lib/docker was a different filesystem. After the merge it is the same one, and a full Docker data-root is a stopped box. The floor sits in captureAllRecoveryUnits, checked BEFORE anything is written: below the reserve, that ONE app's capture is refused, its previous unit is left byte-identical, the R-158 alert fires with the space figures, and the loop continues. Two terms whichever binds first (97% used / 1 GiB free) in fillwatch's shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned before a refusal can happen — a floor that fires before its own warning is a silent failure wearing a threshold. Headroom, never unit size: a per-unit cap would be R-163 rebuilt inside one volume. Refuses, never deletes: nothing here is generational, so pruning could only destroy a different app's only local copy; pruneStalePrimaryDirs is an orphan sweep, not retention, and must not be repurposed. Tests 1184 -> 1191. One fixture strengthened mid-red-proof: the "old 20 G ceiling is gone" test sat at exactly 20 GB and survived a literal UsedGB > 20 cap — hollow. Now 120 GB, and the mutation fails it. |
||
|
admin
|
9a3c4855d7 |
v0.191.2 — a quiet fill check now says so (R-167)
gates / gates (push) Successful in 9s
Earned during v0.191.1's own live validation. After the customer had been warned, a restart produced ZERO fillwatch lines — equally consistent with 'ran and chose silence' and 'never ran'. Proving the checker was alive needed a deliberate crossing into the critical band. For an edge-triggered check the quiet run IS the healthy steady state, so that ambiguity is permanent rather than rare. Check now logs a per-RUN summary on every run, counting unreadable separately from healthy so a drive that has quietly gone unreadable cannot read as 'all fine'. |
||
|
admin
|
cf48214f6c |
v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored. |
||
|
admin
|
582135f861 |
v0.190.0 — the boot settle window, both gates on intent, and R-171
gates / gates (push) Successful in 8s
R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix was written). Replacing isBootOrphan's container-count term with recorded intent made a drive-gate-stopped app read as a boot orphan: the gate stops apps with `compose down` (zero containers) and never touches desired_state, because it is not the customer. Observed on 9201 with the drive held unmounted — the sweep found and started it, burned both attempts, and handed it to the dead-app alarm. The write hazard did not materialise (the unbound mountpoint is host-root-owned and the guest is unprivileged) but that protection is accidental and untested. New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive already refuses this; the sweep bypassed it. R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a fleet docker was still restoring — three of six hard resets. Now a settle-then- sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at the end; ends on settled or a 50s budget, and the log says which. The budget is 50s because settle+budget+one retry must stay under the 90s dead-app grace — a test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn rather than the grace being widened to hide it. Widening the window made two more holders reachable, so the one gate covers all three: an absent drive, a quiesce, and an in-flight app-data operation — reusing quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks(). R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way table; absent keeps the old hasContainers behaviour exactly. Its comment argued for the container count and was rewritten. presentStable is untouched. The two gates' agreement is pinned from both sides against one fixture table. 27/27 packages green; 6 red-proofs observed FAIL then restored. |
||
|
admin
|
dbcb306fcf |
v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b)
gates / gates (push) Successful in 8s
The box stops inferring the customer's intent from a container count and reads
what they actually asked for.
Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).
ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.
Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).
Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.
No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
|
||
|
admin
|
4ed938cce4 |
D5: an app restore works from the drive alone (v0.188.0)
The recovery unit on the customer's drive now carries the PORTABLE secret class, so Tier-1/Tier-2 restore no longer depends on the whole-guest tier. A customer needs the drive and nothing else. Part 0's rulings overturned the brief's recommendation, on evidence: - the data_key flag is untrustworthy (4+ encryption keys the catalog itself labels as such are unflagged) -> R-127 - a DB password is not resettable in practice: POSTGRES_PASSWORD is ignored once PGDATA is non-empty, so a regenerated value leaves the app unable to authenticate against its own restored rows while the dump replay still reports success (proven on a throwaway postgres:16-alpine) Ruling (operator): type:secret travels, type:password never does, minus the nonPortableSecrets code register. Plaintext -- withholding the internet- reachable class is what licenses that, and the two are coupled. Precedence: the UNIT WINS over the guest -- the unit's secrets were captured in the same run as the dumps beside them, so they match the data being restored. The fail-closed data-key gate is unchanged. Secret values are never logged; the manifest records NAMES only. |
||
|
admin
|
2f27a363d5 |
R-108: network storage may not host an app's data namespace (v0.187.0)
This is D5's precondition and it is now met. An app's namespace root IS its backup root: namespaceRoot returns a non-system drive path as-is, so the recovery unit lands at <HDD_PATH>/backups/primary/<stack>/. On a NAS that sits inside the share, which FileBrowser binds WHOLE — share root, :rslave, download:true. The bind was NOT narrowed, and establishing why inverted the fix. The share-root :rslave bind is load-bearing (a 2026-07-22 probe proved an in-container access through it wakes the idle automount trigger), and scoping is undefinable anyway: apps on a share store at <share>/<app>, there is no userdata/ layer, and creating one would write Felhom convention onto a customer's own NAS, which R-67 forbids. So the browsing surface cannot be narrowed and the backup tree must never be placed under it. Operator ruling: refuse the placement, keep the browse bind. Tier 2 already refuses network targets for this reason (F-6C-1). Nothing stranded: zero apps on network storage across all six hub customers including Peti. R-67's browse capability is byte-identical. FIVE surfaces, not the four the register named — settings.RefuseAsAppNamespace is the single predicate. The deploy POST is the real boundary (it accepts any caller-supplied HDD_PATH; DeployStack validates only os.Stat). Surface 4, handleStorageDecommission mode=migrate, guarded only its SOURCE, so a whole namespace could be decommissioned ONTO a NAS — that one is not in the register. Fails closed: /mnt/felhom-drives holds both kinds, Kind exists only on a registered path, so an unregistered path under that root refuses. Supersedes README's "NAS backup locality — decision A" (v0.118.0). 9 tests, all non-effect (nil stackMgr, so a guard that misses panics rather than passing). 4 red-proofs, each mutation asserted to have landed. Suite rc=0, 27 packages, 0 FAIL. vet rc=0. Template + emoji gates OK. |
||
|
admin
|
b331f18424 |
v0.186.0 — R-114 + R-112: tell the truth about the backup target, then show it
Two defects E-2d found on a real box, fixed in this order deliberately: the
message is corrected BEFORE it is put on screen, because switching on a banner
that lies is worse than a silent one.
R-114 — the third state. resolveBackupTargetState had two outcomes: a disk
claims the target (healthy), or nothing does (degraded, "the backup is on the
system disk"). The state "configured, and its drive is gone" had no branch, so
it fell into the second and inherited its message AND its offer. Observed live
with the target detached: degraded:true, target:"felhom-backup" plus the
system-disk copy (false -- the backup was on a drive that had vanished) plus
offer_path naming that same vanished drive as the remedy.
New BackupTargetState.TargetAbsent discriminates. Degraded keeps its meaning
("is there a problem") so the wire contract is unchanged for every consumer;
TargetAbsent answers "which problem", because the two have opposite remedies --
attach any second drive, versus reconnect THAT one. Copy routed through
degradedMessageFor so one place still decides what a customer reads. The offer
is suppressed on the branch itself, NOT left to firstOfferableDrive's
Disconnected skip: that flag is set by the agent-side gate in another repo
(R-113), and this state must be correct independently of it.
R-112 — the state finally has a consumer. The endpoint was byte-correct and
nothing in the product ever asked for it: templates fetch 18 distinct
/api/storage/* endpoints and backup-target[/assign] were the only two with zero
references. Server-rendered on /backups now, following the existing
SingleCopyWarning banner pattern -- not a 19th JS fetch, because a banner that
needs JavaScript to appear is one more thing that can silently not happen.
backupTargetView returns nil for healthy and unknown so those render nothing at
all. The offer control POSTs to the existing assign endpoint behind the standard
inline confirm, never auto-submits, and surfaces restart_required honestly
instead of adding a self-restart.
Scenario E (the seam test) drives backupsHandler over httptest and asserts the
RENDERED HTML -- handler -> view -> resolver -> template. It deliberately does
not call the resolver and assert a string, which would prove the resolver that
was never broken. Deleting the one line that sets data["BackupTarget"]
reproduces the R-112 state and fails every render assertion.
Tests 326 -> 338 (+12) in internal/web; suite green (27 packages); both template
gates pass. Three red-proofs run and reverted, files byte-identical after.
MinAgent unchanged at 0.113.0: R-114 reads BackupTarget/MountPath/GuestPath/Role,
none of which R-113 altered (it changed BoundUnderParent, which this code does
not read). demo-hp on agent 0.113.0 is not held.
The absent copy is verbatim the hub's customerMessages["backup_target_absent"]
so the banner and the email tell one story -- filed as a two-repo drift risk,
not solved.
NOT LIVE-VALIDATED. Scenario C cannot occur on a healthy box; Session C proves it.
|
||
|
admin
|
cdaeb36972 |
v0.185.1 — E-2: the offer endpoints were mounted where nothing routed to them
Registered as /api/backup-target inside ServeStorageAPI, which main.go mounts ONLY at /api/storage/. Live result: endpoint not found, while every unit test passed -- the tests called the handlers directly and never travelled the mount. Caught by the first live call, which is why the live call is part of the procedure. Moved to /api/storage/backup-target[/assign]. A new test asserts the dispatcher source contains both paths, so a handler nothing routes to fails the suite -- the seam-wiring rule applied to a route rather than a button. |
||
|
admin
|
3f7cf2a965 |
v0.185.0 — E-2 Parts 3+4: the offer, and the honest degraded state
The half that makes the rest work: a degraded backup target recorded only in
config is the silent-degradation pattern this arc has spent a week removing.
Part 3 -- POST /api/backup-target/assign moves the target via the agent's
POST /backup/target. It is the ONLY writer of the role: registration does not set
it, the drive-gate does not, no scheduler does. Declining is not calling it. The
agent returns restart_required rather than restarting itself, because restarting
with a backup in flight records a spurious tier failure for a backup that
actually succeeded (E-1 did exactly that).
Part 4 -- GET /api/backup-target returns the state and, when degraded, Hungarian
copy in FACT -> CONSEQUENCE -> REMEDY order, pinned by a test: a customer told
only the fact cannot act on it.
Healthy renders NOTHING -- no badge, no reassurance, no tonal change.
degradedMessageFor is the single decision point, so exactly one place could start
decorating a working box. Red-proofed: reassuring on the healthy branch fails
Scenario E.
UNKNOWN is not degraded: an unreachable or pre-R-82 agent means we could not ask,
which is not evidence of degradation (R-88 Part 2's class).
A HOLLOW TEST caught by its own red-proof: TestUnknownStateRendersNothing used
{Known:false} with Degraded left false, so it passed even with the !Known guard
deleted -- the second condition covered for it. Now {Known:false, Degraded:true},
which fails properly. Without the red-proof the test would have been decoration.
State is derived from the AGENT, never from our intent flag: on the two boxes
migrated by hand in E-1 the intent was never recorded while the drive really is
the target.
MinAgent: 0.113.0
Green gate: build + vet + test rc=0 (27 packages), run separately from this commit.
|
||
|
admin
|
4d6c8a6056 |
v0.184.1 — E-2b keying fix: the backup-target branch was unreachable
Caught before deploy by tracing a.Path back to its source, not by a failure. The 0.184.0 image is superseded and must not be shipped. ReconcileDriveGates resolves the target as isTarget[a.Path], and a.Path is the REGISTERED StoragePath -- for an external drive that is the GUEST path /mnt/felhom-drives/<name>, not the agent's host MountPath (/mnt/<name>) that /disks reports. driveTargetByPath keyed on MountPath alone, so the lookup never matched: every absent drive, the target included, fell through to the generic storage_disconnected. The alarm would have looked wired, passed its own unit tests, shipped, and been silently wrong on exactly the drive it exists for -- the same defect class E-2b was opened to fix, one level down. Now keyed under BOTH paths, mirroring planDriveGates which already registers present[] under GuestPath and MountPath for the same reason. Red-proof: MountPath-only keying fails with "the backup target is not resolvable by its GUEST path -- the gate passes a.Path (the registered StoragePath), so the backup-target branch would never fire". Green gate: build + vet + test rc=0, run separately from this commit. |
||
|
admin
|
c1a63de1c7 |
v0.184.0 — E-2b + Part 5: wire the drive-absent alarm that was never called
NotifyStorageDisconnected and NotifyStorageReconnected were defined and called from NOWHERE. Registered in allowedEventTypes, in DefaultEnabledEvents, and given a Hungarian message on the hub -- and never invoked. A drive going absent produced apps stopped, a WARN log and a UI badge, then silence on every channel. Verified against the gitignored-cmd/ trap with a positive control. Fifth instance of this class, found by E-2 Phase 0 rather than by a failure. A drive that is ONLY a backup target has no apps to stop, so it was silent twice. ReconcileDriveGates now calls both halves. When the absent drive is the whole-guest backup target it raises the more specific backup_target_absent (error) instead -- never both, since two mails for one event trains people to ignore the channel -- and recovers as backup_target_restored (info, the existing pairing-gated pattern; severityNotifies NOT widened). The recovery mirrors the alarm's choice or the operator cannot match them. Which drive is the target comes from the AGENT (/disks backup_target, >= 0.112.0), not from our StoragePath.BackupTarget: that is customer INTENT, and on the two boxes migrated by hand in E-1 the intent was never recorded while the drive really is the target. An older agent omits the field -> false -> generic alarm, never a wrong one. Before this an absent backup target had NO prompt signal: the tier stays DUE (targetStoragePresent checks name presence, never reachability), so the only evidence was its own failure at the next due cycle, up to ~24h away. The R-100 shape. Tests observe the WIRE, not a mock, because the failure class is "nothing arrives": a real Notifier posts to an httptest hub and the test asserts the event type and severity that actually went out. MinAgent: 0.112.0 Green gate: build + vet + test rc=0 (27 packages), run separately from this commit. |
||
|
admin
|
ff058a4f10 |
E-2 Part 1: the backup-target role on StoragePath (foundation, not yet wired)
Foundation only, no version bump -- nothing customer-visible changes yet. The
offer UI (Part 3), the degraded banner (Part 4) and the controller half of the
absent-target signal (Part 5) are NOT here; they are tracked as E-2 in
OPEN-ITEMS.md so this cannot become a sixth "seam built but never wired". The
fifth was found by E-2's own Phase 0: NotifyStorageDisconnected is defined and
called nowhere, so a drive going absent emits no event at all today.
StoragePath gains BackupTarget bool -- the sibling role to Schedulable/
IsDefault/Kind. It is INTENT, not truth: the authority is the agent's
backup.local_backup_target, and truth is read from GET /backup/tiers. This
records what the customer ASSIGNED so the controller can render the state,
notice the drive going absent, and detect drift.
Invariants, each pinned by a test asserting the CONSEQUENCE not the mechanism:
- a drive NEVER acquires the role by appearing (registration does not set it)
- exactly one carrier; assigning moves rather than duplicates
- sticky: a new bigger/faster drive does not steal an assigned target
- an absent target STAYS assigned -- clearing on disconnect would be a silent
retarget by omission ("no target configured" instead of "drive missing")
- a network share is refused (R-108 risk model; the role is about LOCAL disk
failure)
Red-proof C: adding auto-elevation to AddStoragePath fails
TestRegisteringDrivesNeverAssignsTheBackupTarget with
registering drives assigned the backup target "/mnt/hdd_1"
Attributes may suggest and refuse the absurd, never select: demo-felhom's backup
drive is an external USB HDD and BOTH demo boxes report removable=0, so a
transport rule disqualifies the reference drive and a removable rule finds no
candidate at all.
Green gate: build + vet + test rc=0, run separately from this commit.
|
||
|
admin
|
fd50a73e65 |
C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0)
Both are the system reporting healthy while the customer is not, and both live in the same status-derivation code. Neither is fixed by making the system quieter. C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén." — at the moment the customer pressed it because files were missing, with 156 MB of BookStack's data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit). Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable by any customer action. C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no longer contradicts the alarm on the same screen. README's claim that faults "still surface as restarting" was a wish with no test — corrected in place; it is the seventh such instance. Six red-proofs observed, including the one that matters most: adding StateRestarting to IsDownState fails the brief-restart test with "every deploy and update would page the operator". go test ./... rc=0, 27 packages, run and read separately from this commit. |
||
|
admin
|
3f048e042b |
R-101 + F-DIAG: the restore dialog names the last SUCCESSFUL copy (v0.182.0)
Tier2LastRun is the attempt clock and was rendered as 'Legutóbbi másolat' in the restore confirm dialog. New LastSuccess + SuccessTracked anchor; tier2Update makes the three rebuild sites safe by construction. F-DIAG: six distinct causes, target-aware redaction. |
||
|
admin
|
e000e201af |
R-100: record the offsite last-SUCCESS anchor (v0.181.0)
LastRun records an attempt, not a result. New OffboxTarget.LastSuccess, set only on the success branch via the pure offboxAnchorAfterRun rule, carried to the hub as last_success. Closes two silent-wipe sites (settings save, hub re-apply). |
||
|
admin
|
fb91c8d766 |
F-OBS: the dead-app check gets a positive observable (v0.180.0)
deadapp-check had no observable at default info level: its per-cycle line goes through Scheduler.dbg(), gated on logging.level==debug, so on a default box it is never PRODUCED (not merely filtered) and cannot reach the always-DEBUG ring. A 30s interval also puts it on the scheduler's quiet path. 'No alarms' was therefore indistinguishable from 'the detector never ran' — which undermines confidence in the F-CRIT-1 fix in the field. A periodic summary, not a line per run: at 30s a per-run line is 2880 lines/day, which is why the original author chose silence. Every 20th scan (~10 min) emits one INFO with the scan count, apps evaluated and apps down. A test pins the cadence so it cannot be widened into uselessness. Also corrects the 'unquiesce guaranteed by defer' comment — fault 10 established the guarantee is the crash marker plus Recover(). |
||
|
admin
|
079265ad8e |
F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0)
F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
1. restartAll returned nothing, so the failure was logged and dropped and no
caller could learn a customer's app had not come back. It now returns the
stacks that failed; both call sites record the outcome.
2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
means the user stopped it'). The quiesce loop stops stacks by the same
compose-down path, so a failed restart is also StateStopped and was
whitelisted into silence. Loop.FailedRestarts() is now the only thing that
lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).
F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.
Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).
Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
|
||
|
admin
|
8f46495426 |
seam sweep: move the TieredBackend witness into production code (no version bump)
A witness in a _test.go file fires on go test and go vet but NOT on go build alone — and a build-only step is exactly how the R-88 Part 2 near-miss would have shipped. Moved beside the type it pins, and added one for AgentVersionReporter. No defect found: quiesceBackend and *Client both satisfy their interfaces today. No version bump, no deploy — compile-time only. |
||
|
admin
|
86ea482fc1 |
controller v0.178.0 — R-88 Part 2: only a positive 'never' fires the valve
MinAgent: 0.105.0. scheduledRunAllowed fired on any nil age; it now requires a licence from valveLicensed, which grants it for AgeStateAbsent and for a LEGACY agent, and refuses it for AgeStateUnknown. An unreadable storage no longer masquerades as a first-ever backup and no longer quiesces apps outside the window. A missing wire field means legacy, not unknown — deliberately. Treating it as unknown would stop the valve firing on un-upgraded boxes and starve genuinely new ones. Degrade logged once; unrecognised future values also map to legacy. Caught in passing: TieredBackend is satisfied by a RUNTIME assertion, so the signature change compiled and vetted clean while quiesceBackend silently stopped satisfying it — which would have degraded every box to the single-tier path with no error. Added a compile-time witness. Also corrects the notifier comment that claimed operator-only came from a missing customerMessages entry; enforcement is hub-side operatorOnlyEvents (hub 0.79.0). |
||
|
admin
|
e9c99566b0 |
R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import), wired by an init-only setter because main.go builds the notifier after the loop. Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and recovery rides recordSuccess's existing bool. Uses NEW operator-only event types; reusing backup_failed would have emailed the customer in Hungarian about a backup they cannot act on, since it has a customerMessages entry and is in live enabled_events. Requires hub >= v0.78.0. R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE, consumed at the same single derivation point. 180s grace, derived from the deploy flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app that genuinely fails to come back still alarms. |
||
|
admin
|
32200c7b5f |
R-88 Part 1: a failing backup stops re-quiescing (v0.176.0)
internal/quiesce had no failure counter, no backoff and no breaker, and the driver is a plain 5-minute ticker — so a tier that was due and kept failing stopped and restarted every customer app stack every 5 minutes indefinitely. Live on demo-felhom 2026-07-27: three cycles in eleven minutes against an unreachable PBS tier; it ended only because PBS recovered. The breaker gates the QUIESCE, not the backup — the harm was the outage taken to attempt it, so backed-off tiers are dropped before any stack is stopped. Per target (a broken offsite tier must not suppress a healthy local one), 15m→30m→ 1h→2h→4h capped, reset on success, never permanent, never applied to TriggerNow, and stillRunning is not a failure. State is in-memory on purpose: forgetting a backoff costs one attempt; persisting one could outlive the fix. Part 3 invariant recorded on scheduledRunAllowed — a missing value means UNKNOWN, and only a positive 'never' may fire the safety valve. Fourth instance of the class (hub v0.12.0, v0.73.0, R-81, R-88). Part 2 (unknown != never) is NOT in this commit: the agent returns byte-identical responses for 'read errored' and 'never backed up', so the controller cannot tell them apart. That needs an agent wire change and is tracked separately. |
||
 Claude Code
|
9e5ea56853 |
v0.175.0 — R-82: a tier that overruns the quiesce bound defers the rest
Operator ruling 2026-07-26: let the first backup run as long as needed; other backups shouldn't start until finished. A first FULL offsite snapshot runs for hours, far past max_quiesce. When that bound elapses the app resumes (unchanged), but the loop then started the NEXT tier while the first was still uploading. Now it breaks and defers the rest to a later poll — vzdump still holds the guest lock, so the second start would be refused by the agent (409, v0.99.0) or fail on the lock, and a failed backup never satisfies a cadence, so the tier would retry into the same wall forever. pollTier returns (phase, stillRunning, err). The app still resumes exactly once. Red-proof observed and restored; full suite green. |
||
|
Claude Code
|
de96efc0c5 |
v0.174.0 — R-82 Slice B: one quiesce window, two backup tiers
MinAgent UNCHANGED — degrades gracefully against ANY older agent.
The agent gained per-target tiers in v0.97.0. The controller owns quiescing,
so the multi-tier schedule is reconciled here: every due tier is collected up
front and run inside ONE quiesce window (one stop, N sequential backups, one
resume). Two cycles on the weekly night would mean two app outages for one
night's work.
Dedup rule: local-only -> one quiesce; PBS-only -> one quiesce; BOTH due ->
ONE window with both backups inside; neither -> no quiesce.
- quiesce.TieredBackend + BackupTier + ErrTiersUnsupported (optional extension)
- agentapi: BackupTiers/BackupDueFor/StartBackupFor/BackupStatusFor;
targetQuery("") yields an EMPTY suffix so untargeted hits the pre-R-82 route
byte-for-byte
- Loop.resolveDueTiers = the dedup rule in one place, agent order preserved
- quiesceAndPollTiers + pollTier: app stays quiesced until the LAST tier
snapshots (resuming earlier loses app-consistency on the DR tier). Consequence
stated in the docs: both-due-night downtime = first tier's full backup + last
tier's snapshot, which is why tiers run fast-first.
- Manual 'Mentes most' covers EVERY tier, due-ness ignored.
- Window-gate safety valve now uses the OLDEST due tier, so a stale DR tier
cannot be starved by a fresher local one.
Capability detection: /backup/tiers 404 = pre-R-82 agent (the documented
route-probe mechanism). Not a featureProbes row on purpose — the loop needs the
tier LIST, not a yes/no. Degrade logged exactly once per process.
Tests +11, full suite green. Red-proofs #2 and #3 observed and restored.
|
||
|
admin
|
9056f01fae |
v0.173.0 — R-77: endpoint-drift detection, samba protected-set gate, channel log honesty
Source: felhom.eu/documentation/audits/DIAG-agent-channel-2026-07-26.md bootstrap.DetectEndpointDrift names a controller.yaml vs bootstrap.json local_api.endpoint divergence -- one ERROR carrying BOTH values and BOTH paths, its own event type local_api_endpoint_drift, and its own Hungarian banner shown ABOVE the channel banner because drift is the cause and "agent unreachable" the symptom. It writes NOTHING: reconciling from bootstrap.json would clobber a correct controller.yaml on any half-provisioned or hand-repaired guest, so the authority ruling is deferred to R-78. Fail-safe silent on absent/unparseable/ incomplete bootstrap and on an empty endpoint (ensureLocalAPI's fill-if-missing path is untouched). Fingerprint compared as a BOOLEAN only; token never compared, logged or exposed. EffectiveProtected now gates samba on Enabled && UserSet, mirroring BOTH of reconcileSambaAt's early returns, and the doc comment is corrected in the same change -- it claimed "detection and deployment agree in both directions" while citing only !smb.Enabled, an assertion that went false when !smb.UserSet was added. Not over-suppressed: sharing on WITH a password and a dead container still alarms. Channel log: the debounce placeholder is stateUnconfirmed (rendered "unseeded") instead of "up", so a born-down channel no longer logs "up->down" and orUnseeded stops being dead code. Logging only -- the placeholder is still matched in the re-arm condition, so F2 born-down alerting is byte-for-byte unchanged and all nine pre-existing channelhealth tests pass. Tests 951 -> 959, all green. Red-proofs A (both directions), E and F. MinAgent unchanged; felhom-agent untouched. |
||
|
admin
|
8fadbd9891 |
v0.172.0 fixup 2: drop import/* from the carry-list
Found on the demo-hp live leg: with import, import/paperless and import/calibre in the carry-list, the derived skeleton RE-CREATES a per-drive drop-zone on every drive forever — the dead lookalike the canonical root exists to remove, and one that is never backed up (class: excluded). Not a zero-removals violation: nothing deletes what an existing box has. Both demo boxes' old drop-zones were verified to hold zero files before the change. |
||
|
admin
|
4773809334 |
v0.172.0 fixup: EnsureImportRoot must apply the convention to the parent userdata dir
Found on the demo-felhom deploy leg: ensuring only <sysNS>/userdata/import left its parent at 755 root:root, because EnsureUserdataDir MkdirAll's intermediates at plain 0755 and chmods only the leaf. That made the system drive's userdata root the one on the box outside the 2775/gid-1000 convention. |
||
|
admin
|
2958946517 |
v0.172.0 — R-75: canonical import root, catalog-derived skeleton, import surfaces
${IMPORT_PATH} = <system namespace root>/userdata/import — ONE drop-zone per box,
on the system drive, injected at BOTH compose-env builders with NO per-drive
fallback (unresolvable leaves it unset so compose fails loudly rather than
quietly building a second, dead drop-zone).
Third BindRoot (RootImport) + Import list in BackupSpec, extended through
ValidateBackupSpec/ClassifyBinds. Load-bearing: a stale `userdata: import/<app>`
entry against the moved bind would be a WHOLE-BLOCK reject, taking the app's
mandatory hdd classification with it.
Exhaustive-root audit: resolveAbs/structuralGuard/ComputeCaptureSet/
ComputeFabBuckets now take importRoot explicitly (an import bind resolved
against hddPath would name a directory on the wrong drive); unresolvable is
refused loudly into Skipped. GetImportRoot added to both provider interfaces.
Catalog-derived skeleton: UserdataSkeleton() -> UserdataSkeletonCarry() +
BuildUserdataSkeleton(), SORTED. The carry-list makes zero-removals true by
construction (`documents` is in no catalog app but on both boxes) and is the
fresh-box floor. The sort is not tidiness: the naive map-order derivation
measured 20 distinct outputs from 20 identical runs, which with fbNeedsRecreate
is a fleet-wide FileBrowser restart loop.
One authoritative compose parser: ParseComposeUserdataMounts now delegates to
ParseComposeClassifiableBinds. Import root excluded from per-app migration.
Surfaces: FileBrowser /srv/beolvasas source; app-page "Hova tegyem a fajlokat?"
with PathEscape deep links (never QueryEscape) and class-driven copy;
data_paths: annotation with the Fork-3 asymmetry; system-owned beolvasas SMB
share refused server-side at handler AND store, button omitted in template.
Caught on the way: the sharing template's row struct was function-local, so
adding {{if .System}} would have 500'd every share row. ShareRow is now
package-level and the render test uses the handler's own type.
Tests 915 -> 949, all green. MinAgent unchanged.
|
||
|
admin
|
f6a8249593 |
v0.171.0: disk-health card device-model label (pairs with agent v0.95.0)
agentapi.SmartSummary.ModelName mirrors the agent's model_name; the card row label prefers the device model over the raw name/UUID, falling back to Name(+hint) on an old agent. Additive. Test + red-proof (drop fallback -> A4 fails). |
||
|
admin
|
9cc8424954 |
v0.170.0: root → Indítópult (302); Vezérlőpult at /dashboard; CLAUDE.md stale-note fix
/ now 302s to /launcher (ONE canonical URL per page — launcher body never served at /); the Vezérlőpult keeps /dashboard + its nav slot (operator ruling, reversing v0.163.0). Nav: Indítópult active on /launcher, Vezérlőpult href=/dashboard active there. Post-login default (/) and the topbar logo (/) flow through the redirect; login target unchanged. Repointed 2 dashboard-card tests /→/dashboard. Corrected the stale vacation/agent-DOWN note in CLAUDE.md (agent is up at 192.168.0.162:8443). Tests: 302 target+status, /dashboard 200, nav hrefs/active; red-proof verified. |
||
|
admin
|
2487681396 |
style: gofmt normalization — no logic changes
gofmt -w across the controller tree (46 files) so gofmt -l is empty — disarms the
formatting landmine where a targeted edit + accidental gofmt -w swept ~46 unrelated
files. Pure formatting: whitespace + gofmt's optional-semicolon removal in reflowed
inline closures. One doc comment reworded ('' -> 'the empty string') to avoid gofmt's
Go-1.19 doc-comment typographic substitition ('' -> curly quote) muddying its meaning.
No build/vet/test behavior change.
|
||
|
admin
|
dbf631312e |
v0.169.1: disk-health card excludes logical/network storage (pbs/lvm/nfs/cifs)
The agent defaults SMART to UNKNOWN on non-physical targets, so they showed as spurious 'Nincs adat' rows. isPhysicalDisk now excludes those types (card + check). Test strengthened: a PBS/LVM fixture with UNKNOWN SMART must still be excluded. |