11 Commits

Author	SHA1	Message	Date
admin	fcef8e069c	one writer at a time, and a check that can run (R-411, R-408, R-407, R-414, R-412a) ... THE WALK FOUND THREE MORE ENTRY POINTS THAN THE REPORT DID. R-411 named one missing acquireRunning. Fixing it and then pinning the invariant with an AST walk surfaced FOUR in total, all of which issued restic commands with no flag: RestoreOffboxScratch - the reported one OffboxRestorePrepareFull - the SECOND request in the customer's own two-step full-restore flow, and the one that actually shells `restic stats`. The UI reaches it FIRST, so flagging only the restore would have left the collision reachable by the ordinary path. RestoreSharesScratch - R-411's exact shape on the shares tier: unlockStale + resticStep, a live web caller, and its sibling PlaceSharesRestore has always taken the flag. RestoreOffbox - no production caller today, but the same dangerous pattern. Flagged rather than left for a future caller to inherit. OffsiteInventoryList is REGISTERED EXEMPT with its reason: it issues only `restic snapshots --json`, measured on demo-hp 2026-08-31 not to take a lock, and flagging it would make browsing a page refuse during a backup for no safety gain. THE REAL DELIVERABLE IS THE WALK, not the acquire. offbox_integrity.go:28 asserted "Every off-site operation takes acquireRunning" since v0.227.0, nothing checked it, and it was false for months - the ninth instance of this project's most-repeated class. The walk is an AST pass, not strings.Contains, because a commented-out call still contains the string. Red-proofed twice: removing the acquire fails it naming RestoreOffboxScratch; an unregistered fake entry point fails it naming the fake. R-407: "It NEVER writes to the repository" corrected in place, not deleted (R-360's rule). `check` takes a lock - and so does `restic stats`, which is the fact nobody had and the one that made R-411 possible. Both recorded where the next reader will meet them. R-414: the proof could not run at all on a box with no registered drive. Part 2.1's determination came out as neither "missed" nor "deliberate": R-356's own test comments say the scratch resolver "still resolves ... only the DESTINATION moves", so it was OUT OF SCOPE, and it was never ruled out on state-only grounds - the one comment about a systemDataPath fallback belonged to PlaceOffsiteRestore, concerned bulk USERDATA, and R-356 overruled even that. So 07 section 6.3's rule applies and now has a fourth consumer. The fallback is SCOPED, because the two callers ask different questions and one predicate answering both is the R-356 defect itself: a UNIT-ONLY restore may fall back to the system data path (07 section 7 records as FACT that a driveless app's unit already lives there indefinitely, and that the same-device placement is intended); a FULL restore keeps today's refusal, because it pulls bulk userdata onto a state-only tier. And the silence ends either way: a proof that cannot start now records ProofResultCannotRun rather than an Err, so last_proof_result is never ABSENT - absent already means "controller too old", and a second meaning on the same field is the StatsKnown trap one level up. It is recorded WITHOUT advancing per-snapshot due-ness, so the app stays retryable once a drive is registered. R-412 leg 1: a per-app push whose unit carried no dump and no tar now says so, at WARN. Wording only - no guard, and the capture is untouched (08 section 8.2). Leg 2 stays OPEN. 16 new tests, 1689 -> 1705. Full suite 28 packages rc=0, all 13 controller gates OK. Red-proofs run and reverted byte-identical for A3/B1 (twice), C1 and D1.	2026-09-01 10:19:36 +02:00
admin	e43b5ec07d	v0.231.0 - the box proves its own off-site copy still holds something (R-87) ... R-87 re-scoped by its own spike and built as Option C. MinAgent 0.129.0 unchanged. THE QUESTION NOTHING ASKED. The weekly check proves the stored bytes are the bytes we stored; it cannot tell us we stored the WRONG thing. A hollow recovery unit backs up cleanly, checks cleanly at 100 percent depth, restores cleanly and gives the customer nothing back - measured on demo-hp 2026-08-31, 120082104 B to 7036 B in one nightly run recorded as a success (R-403). No tier and no cadence asked it. Now offsite-proof does, nightly, on one app. IT DOES NOT prove a restore puts data back into a running app. That stays drill work and 07 section 8 matrix row 4 is NOT moved. THE ACCEPTANCE RULE HAS TWO PARTS AND THE OBVIOUS ONE IS A TRAP. "Check the unit against its own packing list" PASSES a hollow unit, because a hollow unit declares nothing. So: (1) everything declared is present, AND (2) the manifest declares what the app is supposed to have. Part 2 is the whole value. RED-PROOFED: the naive rule makes the hollow-unit test read verdict "pass". THE EXPECTATION COMES FROM INSIDE THE UNIT, never the live box - the snapshot may predate the app's shape, and GetDockerVolumes describes the running app. Database half is DBServiceNames, the same discriminator RestoreFromRecoveryUnit uses. Volume half is ParseComposeNamedVolumes as an EXISTENCE check, not a name match: tars are <project>_<volume>.tar and ResolveDockerVolumeNames derives the project from the compose file's parent dir, which inside a unit is the literal string "compose". Measured on all eight real units on demo-hp the counts match exactly and the naming held every time - but "held on eight" is not "derivable" (R-355). Half a rule that is true beats a whole rule that is invented. THREE OUTCOMES: pass, fail (readable and empty), cannot judge. An app that legitimately has neither a database nor volumes PASSES. RED-PROOFED: alarming on any empty unit makes that test read verdict "fail". IT NEVER WRITES TO THE REPOSITORY and that is asserted on the ARGV as a non-effect: --no-lock, no unlockStale, and m.runner() rather than resticStep so the unlock --remove-all escalation is unreachable. RED-PROOFED: routing it the customer path's way makes the test fail on "unlock" appearing in the argv. IT TAKES acquireRunning ITSELF and skips rather than waits, because RestoreOffboxScratch does not take it (R-408) while offbox_integrity.go states that invariant as universal. DUE-NESS IS PER SNAPSHOT (R-86's model), never per clock. RED-PROOFED: recording a timestamp fails the stored-value test AND breaks the rotation - night 2 re-picks night 1's app. ITS SCRATCH IS A SEPARATE ROOT (backups/offsite-proof) and that is a safety decision, not tidiness: the job deletes its copy on every path, and sharing backups/offsite-restore/<app> would mean a nightly background job deleting the verification copy a CUSTOMER is looking at. It is also invisible to placement, so a proof copy can never be pushed into a live app. SHARED RATHER THAN FORKED: offboxScratchDirIn parameterises the scratch resolver on its ROOT builder, and unitOnlyHeadroom extracts the free-space gate, so the customer path and the proof refuse at the same floor with the same Hungarian sentence. RestoreOffboxScratch's behaviour is unchanged. NEW EVENT offsite_proof_empty, severity error, operator-only - deliberately NOT backup_integrity_failed, whose hub template says the store is DAMAGED. Here the store is sound and the content is absent: different cause, different action. The hub half shipped FIRST, in felhom.eu 1aeaa30 (hub v0.110.0, live and verified), because an unallowlisted type is 400'd and vanishes. 33 new tests, all groups green; full suite 1689 tests, 28 packages, rc=0. All 13 controller gates OK. Five red-proofs run and recorded in REPORT.md. A golden carrying 0.231.0 is OWED - the fleet is on 0.230.0. Viktor's call (R-242).	2026-08-31 20:55:34 +02:00
admin	b8af72764d	R-353/R-357/R-358/R-360: the restore tells the truth (v0.226.0) ... Four defects on the restore surface, all proven on demo-hp during the 2026-08-21 backup-truth drill, all still in shipped code. They share one acceptance idea: a restore surface must state what it actually did, and must refuse what it cannot do. VERSION NOTE. The task specifying this targeted v0.224.0 against baseline f8c9390. Both were consumed earlier the same day by R-330 (0.224.0) and R-331 (0.225.0). Drift re-confirmed against live Gitea before the first edit, operator authorised proceeding, every symbol the spec named re-verified present at the real baseline e5eee50. R-353 -- a restore that gave back nothing still said it worked. RestoreFromRecoveryUnit returned only error, so the surface printed "<app> visszaallitva (<snapshot>)." -- equally true of a run that returned an entire dataset and one that returned nothing. The count already existed and was discarded one line deep: restoreDockerVolumesFrom always returned it, the wrapper threw it away. Now (UnitRestoreResult, error), carrying replayed counts AND what the manifest LISTED, because zero-replayed has two causes that are opposite news. Three cases, three sentences, and EVERY one is a claim about the BACKUP, never about the app -- this path has no SafetyDump discriminator, and 07-backup-architecture 6.3 records that an absent dump says nothing about the app (R-361 destroyed canonical .sql files for four months). R-357 -- the destructive restore had no free-space gate. offbox_reconstitute.go contained ZERO references to offboxFree; all three existing gates guard non-destructive paths. The gate now sits before mapOffsiteRestorePaths, writeSafetyDump and StopStack, so a refusal costs nothing. Position IS the fix, which is why the test asserts StopStack was never called. No headroom multiplier (matches PlaceOffsiteRestore; the x1.1 elsewhere predicts a download). Fail closed on either probe <= 0 -- otherwise `free < need` with need==0 is FALSE and an unmeasurable scratch sails through: a gate present and inert. R-358 -- a failed download was offered as a good one. The gate answered "the directory exists and is non-empty", which is exactly what a part-way restic run leaves. Now a completion marker written 0600 atomically AFTER restic returns nil, with any stale one cleared BEFORE it starts; both orders pinned by an AST test because resticStep is not a seam. Both handlers refuse server-side: the wizard flags control a button, and a hidden button is not a guard. SCENARIO F ANSWERED, and worse than the question assumed: a unit-only scratch IS reachable through the real flow, by the most ordinary route. "Ellenorzo visszaallitas" (mode=unit, advertised non-destructive) writes the SAME directory -- offboxRestoreScratchDir ignores `full` and --include limits what restic extracts, never where -- so a customer who ran the SAFE restore was then offered the destructive one over a unit-only copy. Filed R-396; the marker closes it. R-360 -- the delete refused only while a BACKUP ran. IsRunning() is FALSE for the whole of a verification restore; the five sibling handlers all use restoreOpBlocked(). Its doc comment claimed it already did this, which is why nobody looked -- corrected in place. Red-proofs, each printing the pre-fix behaviour, in CHANGELOG and REPORT. The first R-357 red-proof exposed a hollow test OF MY OWN and it is recorded rather than quietly fixed: the fixture refused earlier at the placement stat pre-pass, so `stops == 0` passed against the pre-fix code. Fixture corrected, assertions reordered so a removed gate reports the outage rather than "no error returned". Green gate clean: 28 packages, rc 0. All 12 controller gates OK.	2026-08-30 19:31:31 +02:00
admin	2c724c9283	R-379/R-380: put the customer's undo copy back when a database restore fails ... R-379 and R-380 were one failure. Both ended with a half-restored database; the only difference was whether it looked broken. Postgres emptied and crash-looped; MariaDB applied part of the dump and reported health=healthy with a zero-row schema-version table. Measured live on demo-hp 2026-08-22. The undo copy was already taken and already good - proven by hand that day on both engines. Nothing in the product could apply it. Now it does, with the same ImportDump call, before any restart and inside the DB-only window. The WHOLE undo set, matched on this run's stamp. writeSafetyDump returned one path for an app with two databases; a rollback on that would restore one and leave the other half-written. When the rollback also fails the app is HELD STOPPED (operator ruling): a running app on a half-written database lets the customer make the damage permanent. Every start path refuses it - customer button, appstop Recover, boot sweep - via the shared driveStartGate, checked ABOVE its driveless early return because these apps have no drive. The marker is ended so nothing auto-restarts it. The row goes red. Cleared with --clear-restore-hold, an operator CLI route. --single-transaction is a belt on Postgres only; MariaDB DDL is not transactional and that is why the rollback is the fix. R-381: the engine's stderr stops reaching the customer (615 bytes on MariaDB, its middle rows out of their own database) and starts reaching the operator log, which never had it. R-382: the summary log prints the volume count it already held. Undo copies resolve to their own app, are marked IsUndo, and are capped at 3 per app, pruned from the capture side. The reported render-as-an-app symptom did NOT reproduce - the live page was read first and had zero occurrences. Tests 1468 -> 1483. Eight red-proofs; ONE PASSED and is reported: the R-381 behavioural test injected below ImportDump. A guard at that layer now convicts.	2026-08-22 18:03:18 +02:00
admin	08eb1a6e3a	R-356: the off-site restore refused every app that has no data drive ... ReconstituteFromOffsite and PlaceOffsiteRestore both resolved the restore destination with the RAW HDD_PATH and read an empty answer as "the app is not installed". For 40 of the 53 catalogue apps that answer is correctly empty and permanent, so both actions refused forever for a running, healthy app — and told the customer to reinstall it "in the same place", which those apps never offer. Separate the two questions. "Installed?" is asked of ListDeployedStacks via a new Manager.isStackDeployed that fails CLOSED on a nil provider. "Where?" is answered by GetAppDrivePath — the same resolver CaptureRecoveryUnit wrote the snapshot with, so the restore aims at the place the backup came from. The 13 drive apps are unchanged: own drive, mismatch check, ack still required. A third refusal, with its own sentence, covers installed-but-no-resolvable-root. Fixtures that marked an app "installed" by giving it an HDD path now state deployment as its own fact. No assertion weakened.	2026-08-22 13:08:46 +02:00
admin	8dbbc98ff2	v0.207.0 — R-249: the retrieval passphrase leaves the page body; R-252/R-253: two refusals learn to say what to do ... R-249. settings_security.html rendered the passphrase into a display:none span behind a Megjelenit button. That toggle stops a browser DRAWING the value and nothing else — the plaintext was in the response body of every render, so a curl of the page returned it. Found by exactly that: it landed in a session transcript while driving the documented rebuild path. The codebase already stated this rule for the recovery code and this page did not follow it (escrow_handlers.go: 'reveal (claim XHR only — R is NEVER templated server-side into HTML)'). The page now carries only HasRetrievalPassword; the value comes from POST /settings/retrieval-password/reveal — CSRF-covered because POST, no-store, and LOGGED as an act, which reading it off the markup never was. The tests assert the RAW RESPONSE BODY. Every test that asked what the customer sees passed while the bytes carried the secret; that is why this survived. Census: the render-then-hide pattern appears twice more — app_info.html (a real per-install app password in a hidden span) and deploy.html. Filed as R-254, NOT fixed here. R-252. A rebuilt box keeps its drives but loses their REGISTRATION. The restore page now states that before the customer presses anything, says the backups and drives are both still there, and links to Tarhely > Meghajtok. Page and resolver ask ONE question — HasRestoreDestination() reads the same GetSchedulableStoragePaths() the scratch resolver reads. R-253. The list promised 'a visszaallitas elobb ujratelepiti' three lines above a refusal that fired BECAUSE the app was not installed. The promise was the wrong half: reconstitution writes to the app's own GetStackHDDPath, which exists only once the CUSTOMER has chosen a drive at deploy time. Auto-reinstalling would mean the product making that choice for them. Copy now says to install first and routes to /stacks/<app>/deploy. Both notices are conditional — a healthy box renders as before, pinned by a test that fails if either becomes unconditional.	2026-08-07 18:04:26 +02:00
admin	062357f778	v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44) ... Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11 immich photos to test offsite restore; both runs flashed success and the photos stayed gone. Two independent defects. R-43 — no offsite path could restore a database. All three buttons were file-only: the two "visszaállítás" actions staged to a scratch folder and never touched postgres, and place-to-live merged only MISSING files. For a DB-indexed app the bytes returned and the app still could not see them. The dump was carried INTO every snapshot and could never be replayed OUT of one. New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop → files overwritten to the snapshot version → start → the snapshot's own dump replayed → health wait. Two invariants: - nothing is ever deleted (-a, no --ignore-existing, no --delete): a file created after the snapshot survives as an extra; - the undo exists before the act — the pre-restore- dump is verified ON DISK before anything is stopped, overwritten or replayed; if it cannot be taken the operation refuses with zero changes. The replay reads the SCRATCH unit: the live unit is never overwritten, so replaying from it would replay the current DB over itself and restore nothing. R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's predated the customer's account by four hours and probed to asset:0/user:0/ album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual AND nightly, now refreshes dumps + units BEFORE capturing. Order is the mechanism: the gap can only ADD files the DB does not reference yet, never remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is verifiable at restore time rather than assumed; the periodic refresh carries a prior stamp forward and never invents one. Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs report their skew, ValidateDump gained an EXACT-match accounts-table sniff for customer-empty dumps, the completion flash states an outcome instead of a mechanism, and the missing-only button now says what it does NOT do. 11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses rather than confirming strength — the first undo mutation was caught by a second guard, and the first table-matching test did not discriminate between the two matchers at all. Both tests were rewritten to the cases that separate them. NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota math, tier-2, and v0.147.x progress semantics beyond one added phase line. Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row stays MISSING, R-3 stays DRAFT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s	2026-07-19 12:21:16 +02:00
admin	b5d78d1e0f	v0.147.0 — feedback slice 1: pressing a button says something ... The systemic complaint, twice in one evening: you press a button and nothing happens. No progress, no ETA, no named result. Three worst offenders, fixed on the two patterns already here (deploy 3-step panel, storage-init status poll). No new framework — that is a ROADMAP item; three targeted cards ship tonight. 4a — a verification restore names its result. The flash said the app had been restored "to a verification folder on the drive"; which folder, on which drive, was invisible, so the customer could not go and look at what they had just asked for. Full path now. The restore page gained a listing of existing verification copies (app, size, date, path) — nothing anywhere showed these, so they piled up and the only way to find them was SSH — each with a double-confirmed delete. That delete is the only one this release adds, so it names a STACK, never a path: the Manager resolves the name inside a backups/offsite-restore root it computed itself and refuses anything landing outside. Red-proofed — neutralise the name guard and stack:"" resolves to the offsite-restore ROOT and takes every copy with it. Refusals are asserted as non-effects. 4b — Megosztás enable shows what it is waiting for. Enabling ran ReconcileSamba synchronously inside the POST handler; on a golden without felhom-samba baked that is compose pulling ~100MB, i.e. minutes of an apparently-hung form post followed by "Beállítás mentve." whether or not anything came up. Detached + polled now, distinguishing "képfájl letöltése" from "indítás" — decided BEFORE the work starts, since afterwards the image is always present. Success is probed, not inferred (compose up -d exits 0 on a crash-loop). The password form starts the same job: with UserSet false reconcile deploys nothing, so on a fresh box that is where the pull actually happens. 4c — "Távoli mentés most" streams real progress. restic was already reporting bytes and percent; the runner seam used CombinedOutput() and discarded them. The manual run now passes --json and scans stdout line-by-line: total bytes, percent, current app. Manual only — the nightly stays silent, pinned by a test that fails if it ever passes --json. The poll now arms unconditionally, closing a race the manual trigger always ran: the redirect rendered before the goroutine wrote LastStatus=running, so the poll never armed and the page sat static during the very run just started. Red-proofed twice. Also closes the golden/controller infra-image drift at the source: infra.Images() derives from the existing pins and --print-infra-images exposes it, so the golden bake can stop carrying its own copy. That copy had already drifted — felhom-samba was never added, so the golden baked 3 of 4, which is why enabling Megosztás pulled at runtime in the first place. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE	2026-07-19 09:30:30 +02:00
admin	3603d1fc7f	Tier-2 engine rework: class-driven legs, v2 layout, NAS-target exclusion (Task 3b, v0.135.0) ... tier2_capture.go: classified apps get TierSecondary per-bind legs (paperless copy shrinks — export drops); legacy apps keep the byte-identical resolver set. v2 relpath-mirroring layout (backups/secondary/<stack>/{marker LAST, recovery-unit/, hdd/<rel>/, userdata/<rel>/}); N>1 native (errTier2MultiDir/tier2AppDataName deleted). Migration=delete-and-rebuild + reconcile; all RemoveAll via tier2SafeRemove (refuses outside backups/secondary/). SSD=state-only tier. selectTier2Target never picks network storage (pinned+auto, F-6C-1). Restore reads v2 behind a marker gate. Part 0: offbox_enlarge_blocked is a persisted one-time Load seed (opt-out sticks), not a getter append. Part 0.5: offsite restore scratch prefers a local (non-network) path. Full v2 test suite + all 10 §10 red-proofs verified. Destructive writes bounded to backups/secondary/.	2026-07-15 10:10:20 +02:00
admin	0cfcc42464	Placement hardening (F-3a-1..4) + enlarge-blocked delivery chain (Task 3a-fix, v0.134.1) ... PlaceOffsiteRestore: live target via raw GetStackHDDPath not AppNamespaceRoot (F-3a-1a: no SSD merge; undeployed refused), placement headroom gate (F-3a-1b), stat pre-pass over all placements before any copy (F-3a-4: no partial writes), scratch removed on success/kept on failure (F-3a-2). mapOffsiteRestorePaths refuses the namespace root itself (F-3a-3). Delivery chain: DefaultEnabledEvents + GetNotificationPrefs append-if-absent migration + settings checkbox + handler slice; paired with hub v0.55.0 allowlist (no customerMessages entry — raw dynamic message survives). +8 tests; all 6 controller §10 red-proofs verified.	2026-07-15 07:54:29 +02:00
admin	2d20859858	Offsite tier policy engine: mandatory userdata, raw-data quota, restore rework (Task 3a, v0.134.0) ... Each toggled app's offsite push = one multi-path restic snapshot (recovery unit + TierOffsite mandatory userdata via ComputeCaptureSet); legacy/undeployed stay unit-only. Loud capture gaps (SP-3.4: restic 0.14.0 silently skips missing paths). Quota = stats --mode raw-data (SP-1; displayed size drops once). Pre-push enlargement gate blocks the userdata enlargement over-quota (unit-only push continues; EnlargedBlocked; edge-triggered notify). forget --group-by host,tags on both sites (SP-2). Restore reworked: scratch off the rootfs + headroom gate (F-A1), unit-only default via --include, size-first full, place-to-live missing-only merge (never --delete). UI: unit/full-two-step/place actions + per-app blocked note; route POST /backup/offbox/place. HUB FLAG: offbox_enlarge_blocked event needs hub allowlist for push delivery. +13 tests; all 10 §10 red-proofs verified. No tier-2/.fab/hub/agent changes.	2026-07-14 22:51:54 +02:00