1e216d34682321aa6007f9422743048709ef3195
85 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 admin
|
6be6c53e29 |
v0.283.0: apps go off-site by themselves (decision 50) with a size warning; a Stop holds during a backup (R-721); page slips (R-724/R-725)
gates / gates (push) Successful in 25s
Red-proofs RP31-RP38. MinAgent 0.131.0 (unchanged). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
0c702f834a |
v0.279.0: after_install (decision 45), known default logins on the page, Part D empty-backup alarm, night chain (R-705), R-706
gates / gates (push) Successful in 27s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
6479933e8e |
v0.277.0: kept data loads from the off-site copy too (R-691 (2))
gates / gates (push) Successful in 25s
Use my kept data / Load consider the off-site snapshot when it is newer than every local copy or the only one; the unit is downloaded alone, judged (drive, data, recorded data version) and only then restored. The page names the copy and its date. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
b6810f14ff |
v0.275.0: a backup's data and its version travel together (R-696, 07 §6.6, D4 option A); R-695, R-691, R-694
gates / gates (push) Successful in 23s
The unit's data files are stamped with the versions that wrote them; the capture keeps the definition the data belongs to; a restore never starts data under another version's definition (unit restores refuse a mismatch; the off-site restore writes the snapshot's definition); every tier's time is its data's; the conversion-copy release needs a dump on the new engine. File-browser sync single-flight + no empty kept folder (R-695); the kept view joins the folder's owning group, language switch resyncs (R-691); a restore-generated login is not shown as the password (R-694). Red-proofs in felhom.eu/documentation/audits/version-travel-2026-09-26/. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
44ae4dea70 |
controller v0.272.0: the backup page says when a whole-box backup does not fit (R-685); R-671, R-670, R-677
gates / gates (push) Successful in 26s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
3c6b49b31c |
controller v0.269.0: whole restore from the second drive; crash loops stopped; exact image digests; steps judged by their own .felhom.yml (decisions 26-28, R-661 R-666 R-667 R-668 R-664 R-665 R-662, 09 6.4 part 6)
gates / gates (push) Successful in 27s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
80e6ad8c47 |
controller v0.267.0: tests off DooPlex's Docker, cut-off copies refused, two pages true
gates / gates (push) Successful in 26s
R-650: internal/dockerexec — every docker exec routed through it; under go test a real docker is refused (opt-in FELHOM_TEST_REAL_DOCKER=1; a stub under the temp dir is allowed). api/stacks/web tests run under a silent stub (TestMain). TestR650_NoBareDockerExec pins it repo-wide. R-640: a dump without its engine's completion marker is refused before the first mutation (unit + off-site restore) and again before any load. R-499: the Tier-2 page's system-disk sentence has four true branches. R-518: the backup button states the measured ~8 min stop. R-626: measured on 9202, not reproduced. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
0054d4bd69 |
controller v0.265.0: R-634 cause fixed, held apps say so, OOM storm alarm, R-647 leftovers
gates / gates (push) Successful in 27s
R-634: a whole-box backup no longer stops/restarts a DEPLOYING app (the measured cause of containers running under 'not deployed'); StopStack and StartStack refuse a deploying stack for every caller. R-625: held badge 'Stopped - restore needed', no Update button. R-636: kernel oom_kill counter; 20+ in 30 min -> one app_oom_storm. R-647: held error per reader, copy_holds key, two log wordings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
48f3336956 |
v0.254.0 — the saved notes follow the language, and the switch becomes a globe (R-557 slice 2 release C; SLICE 2 CLOSED)
gates / gates (push) Successful in 23s
The notes a background run SAVES — last night's backup line, the last error, the proof result, the restore outcome — are written in the BOX's language at the moment they are written. A household that switches sees the previous run's note in the old language until the next run rewrites it: the operator's §16 option 1, stated rather than hidden. EndRestoreOp no longer receives a Hungarian literal from anywhere. The language switch is a globe. Two text links wrapped in the sidebar footer and asked the reader to recognise "Magyar"/"English" as links; a globe is the one symbol every web user already reads as "language", so nobody has to read Hungarian to escape Hungarian. It is <details>/<summary> — a menu with no script, drawn inline because the icon sprite lives only in layout.html and the visitor pages have their own shell. Those visitor pages get the same globe, and a visitor's choice stays theirs: a display-only felhom_lang cookie that langFor reads ONLY when there is no session. A signed-in household can never inherit a language a previous visitor picked in the same browser. POST /lang is CSRF-exempt for a narrow reason written at the exemption — its only achievable effect is the language of the page the victim's own browser shows them — and safeBackPath refuses //evil.example as well as https://, because "starts with /" alone is not the test. §16 taken: a successful claim carries the cookie into the household's setting. TWO PARITY EXCEPTIONS, MEASURED: 106 fixtures compared with a real diff — exactly two change shapes (the dashboard footer, the globe in the shells) and 5 byte-identical, which are the three pages that must not change. I INTRODUCED A DEADLOCK AND THE SUITE CAUGHT IT BY HANGING. UpdateOffboxStatus holds the settings write lock while running its callback; boxLang() wants the read lock; sync.RWMutex is not reentrant. On a real box an off-site run would have hung forever HOLDING the settings lock. Fixed by resolving the language before the callback, and guarded by a test that names the file and line in a second instead of hanging for 25 minutes. MinAgent: 0.131.0 (unchanged). No hub release needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
0fe315b759 |
v0.246.0: an interrupted restore is told; the recovery-code reminder waits until the box can take it
gates / gates (push) Successful in 15s
MinAgent: 0.131.0 (unchanged). Requires hub v0.117.0 for restore_interrupted. R-550 (operator ruling: fix). A design reversed and recorded: the restore op-status was in memory by choice. Now restore-status.json in DataDir, written atomically at both ends of an op. At startup a record still marked running becomes a failed, interrupted result kept per app until that app's next restore, shown on /backups/restore and the off-site wizard, and raised once as restore_interrupted. Cooldowns stay in memory. R-546. The R-543 reminder bar consults the agent's own preflight ok (every blocking item, not a copy of pbs_storage_id), cached 60 s, probed only while paused. /backup/escrow shows a waiting card that polls and reloads instead of red crosses and English diagnostics. POST /api/escrow/start refuses 409 before staging or starting - the direct path chaos night used. Unknown readiness keeps the bar. Red-proofs (each seen failing): restore record across restart; main() calls both startup functions; startup helper with loading skipped; restore page card; bar held back; waiting card; start refusal. go build/vet/test ./... green, 28 packages; controller_gates --fast all OK. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
bdcbd50b42 |
controller v0.240.0: seven defects from the any-tier proof and the first nightly rotation
gates / gates (push) Successful in 13s
R-486 (P1): removing an app with its backups KEPT keeps its Tier-2 record, so the second-drive restore is no longer refused over an intact mirror. R-484: postgis/pgvector/timescaledb images are Postgres (logical dumps). R-485: the backup card sizes the recovery unit and the mirror(s). R-480: a held update's sentence leaves the card once the hold is lifted. R-477: the update's off-site lookup is one snapshots call, no stats. R-478: a copy older than this install's deploy does not count. R-474: "delete backups" deletes the unit, the mirror(s) and the prefs. Tests and red-proofs per row; evidence in felhom.eu documentation/audits/v0240-2026-09-13/ and nightly-2026-09-13-adventurelog/. |
||
|
admin
|
b93c1543da |
controller v0.239.0: any backup tier lets an app update (R-475)
gates / gates (push) Successful in 14s
Operator ruling 2026-09-13. The update precondition walks Tier 2, Tier 1 (own recovery unit, "helyi") and Tier 3 (off-site, 15 s bound; unreachable counts as absent with a WARN) and leans on the first FRESH copy; the backup_max_age rule applies to whichever tier is chosen. No copy anywhere: back up first. Refused only when nothing exists and no backup can be taken. RunAppBackupNow tolerates a Tier-2 failure (WARN) and marks the captured unit proven current. The hold names the tier (második meghajtó / saját meghajtó / távoli mentés) and the date; pre-v0.239.0 holds keep their text. A successful off-site restore now lifts an update hold. The backups page still uses Tier2UnitRestorePoint unchanged. Scenarios G-M tested; red-proofs M, L, the tail and the off-site clear in felhom.eu documentation/audits/rulings-r472-r475-2026-09-13/. |
||
|
admin
|
cbcca03061 |
v0.238.1: the nightly backup leaves an app alone WHILE it is being updated, not only once it is held (slice 4 follow-up)
gates / gates (push) Successful in 13s
Found live in v0.238.0 Scenario F on demo-hp: during an update's 5-minute health wait the app is not yet held, and the periodic recovery-unit capture at 10:17:09 wrote the never-started definition (alpine:3.20) into its PRIMARY unit, 53 s before the hold landed. The Tier-2 mirror the hold names survived only because Tier 2 runs daily; a nightly Tier 2 inside a verify window would have mirrored the broken definition over the copy the customer is told to restore from. backup.Manager.isHeld — consulted by the capture sweep, the Tier-2 run and the volume dump — is now also true while a guarded update is moving the app, via SetUpdatingCheck wired in main.go to stacks.Manager.IsUpdating. Test with positive control + red-proof; wiring pinned. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS |
||
|
admin
|
0d402f711d |
v0.237.0: the Update button takes a backup first, and tells the truth (update arc slice 4 — R-448, R-443, R-439)
gates / gates (push) Successful in 13s
POST /api/stacks/{name}/update is now a guarded job answering 202:
cheap refusals (hold — R-439, busy, migration, deploying, memory via the
deploy's own memoryVerdict, a fixed 2 GB disk floor, and no restorable
Tier-2 copy) → backup-first when the proven copy is older than
update.backup_max_age (24h) → safety dump BEFORE the pin moves → pin →
pull (failure puts the pin back) → up → health (.felhom.yml check or 60 s
settle, update.health_timeout 5m). Not healthy → the app is stopped and
HELD (RestoreHold reason update_failed, same store and gate as R-379) and
the page names the backup to restore from; the pin stays. Success is only
ever update_phase=done after health (R-443). UpdateStack is deleted.
The restorable-unit predicate is EXTRACTED to backup.Tier2UnitRestorePoint
and shared with the backups page (row pinned unchanged). The copy is aged
by the last successful Tier-2 copy, not the manifest created_at — measured
on demo-hp that created_at moves only on definition changes.
Crash safety: update-journal.json before each phase; RecoverUpdates before
the boot sweep, ResumeInterruptedUpdates after the guards are wired.
Three unattended start paths ignored a hold and now honour it: the
drive-return gate (restart + boot recreate) and the nightly volume dump.
The nightly capture and Tier-2 run skip held apps so the restore point
survives. No automatic rollback — measured per-app; route back = restore.
Tests A–H across stacks/backup/api/web/cmd; six red-proofs seen to fail.
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
|
||
|
admin
|
2358e561b7 |
R-403: a poorer copy must never delete a richer one
gates / gates (push) Successful in 11s
MEASURED FIRST, then fixed. On the shipped v0.229.0, on demo-hp, an app's Tier-2 copy went from 120 082 104 B (4 database dumps + 3 named-volume tars) to 7 036 B (none of either) in ONE nightly run, and the run recorded itself a success: 'Tier 2 copied docmost -> ... (14.9 KB, 0 leg(s), 0s)'. Evidence: felhom.eu/documentation/audits/DRILL-r403-tier2-delete-2026-08-31/. The mechanism was three individually-correct lines: RunTier2 guards the unit leg with os.Stat only (does the folder exist), rsyncMirror is rsync -a --delete, and nothing between them compared source to destination. An EMPTY unit is a folder that exists. THE GUARD. One predicate, unitCarriesData/unitIsHollow (r403_hollow.go), asking the MANIFEST and never the byte size - a big compose tree with no dumps is dangerous, a tiny unit for a tiny app is fine. Fail closed on an absent or unparseable manifest. RunTier2 skips the unit leg when the source is hollow AND the destination is not; the other legs still run, the run is not failed, and the skip is recorded for the SURFACE (CrossDriveBackup.UnitLegSkipped + UnitPackageDate) as well as logged. --delete STAYS and shrinking stays legal. 07 section 8 row 5's derived-copy rule is unchanged; the fence is exactly one shape. TestR403_DataLegShrinkIsUnaffected is the guard on the guard. THE HONESTY. A preserved package is older than the run that preserved it, so the card carries a notice and the unit-restore confirm names the PACKAGE's date - read from the mirrored manifest's own created_at, not from the status record - plus a clause saying why it is older. THE CAUSE. RestoreTier2Unit now refills a hollow or absent primary unit from the mirror it just restored from, INSIDE the call before returning. The hollow manifest was written two seconds after a restore by the 5-minute capture job; any follow-up job races it. The capture itself is NOT guarded: a capture describing an empty drive as empty is correct, and with the primary refilled there is no hollow state left to describe. Never over a complete primary, never after a failed restore. recordTier2Success and tier2UnitConfirmMsg keep their old signatures as thin callers, so no existing test needed editing. New seam unitRehydrate, separate from tier2Mirror on purpose. 22 new Go tests. Red-proofs run and reverted: A6 (predicate -> size threshold), B1 (guard removed -> the copy's 3 files are DELETED and the seam is called), B6 (a general never-shrink rule -> the shrink case fails), C2 (only-when-hollow dropped -> the complete primary is overwritten). |
||
|
admin
|
b8af72764d |
R-353/R-357/R-358/R-360: the restore tells the truth (v0.226.0)
gates / gates (push) Successful in 11s
Four defects on the restore surface, all proven on demo-hp during the 2026-08-21 backup-truth drill, all still in shipped code. They share one acceptance idea: a restore surface must state what it actually did, and must refuse what it cannot do. VERSION NOTE. The task specifying this targeted v0.224.0 against baseline |
||
|
admin
|
92cebb8c95 |
R-330: stop the backup alarming about the apps it is holding down (v0.224.0)
gates / gates (push) Successful in 11s
Measured live on demo-hp 2026-08-30 (controller 0.223.0): the nightly db-dump
and offbox-backup legs stop each stack ~13s to tar its volumes while the
deadapp-check job scans every 30s, so the scan caught whichever stack was
mid-cycle and pushed app_start_failed to the customer. 61 e-mails about apps
that were never broken.
The defect is not a missing mechanism. quiesce/suppress.go solved exactly this
in v0.179.0 and works -- but classifyRunStates read only the quiesce loop's set,
and that loop covers the WHOLE-GUEST backup. The per-app legs stop stacks
through Manager.DumpAppVolumesSafe, which registered with nothing. Two
mechanisms stop apps on purpose; only one told the alarm. Fifth instance of the
"seam built but never wired" class, and the first where the unwired half was a
consumer.
The suppression now rides AppStopGuard, which already brackets every deliberate
stop in the product (Begin before the stop, End after a successful restart) at
all three call sites, and which main.go hands as ONE object to the backup
manager and the exporter. scanDeployedAppRunStates takes the union of both sets.
All three per-app stop paths are covered, not only the reported nightly one.
It cannot latch -- End() runs only on a restart that SUCCEEDED, so unlike the
quiesce loop an open-ended hold is a real hazard here:
1. ReleaseFailed drops the entry IMMEDIATELY on a restart that broke, wired at
every failure path, so the app alarms on the next scan;
2. Begin REPLACES the set (one marker file = one operation);
3. appStopMaxHold (6h) caps a hold nothing released, logged at WARN.
Grace is 180s, deliberately quiesce's own constant and derivation. Suppression
is NOT persisted: after a crash the guard holds nothing and a down app must
alarm. ReleaseFailed keeps the durable crash marker; a test pins that.
Three companion red-proofs, each printing the pre-fix value (REPORT.md section 5):
- drop markStopped from Begin -> "suppressed at stop = map[]"
- drop ReleaseFailed from the dump -> "map[bookstack:true] after a restart that FAILED"
- pass nil instead of appStopGuard -> the AST wiring test fails
The third is load-bearing: the component was never the broken part, so a suite
that only injected it would have been green against the shipped defect.
Green gate clean: go build + go vet + go test ./... -- 28 packages, rc 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM
|
||
|
admin
|
2c724c9283 |
R-379/R-380: put the customer's undo copy back when a database restore fails
gates / gates (push) Successful in 11s
R-379 and R-380 were one failure. Both ended with a half-restored database; the only difference was whether it looked broken. Postgres emptied and crash-looped; MariaDB applied part of the dump and reported health=healthy with a zero-row schema-version table. Measured live on demo-hp 2026-08-22. The undo copy was already taken and already good - proven by hand that day on both engines. Nothing in the product could apply it. Now it does, with the same ImportDump call, before any restart and inside the DB-only window. The WHOLE undo set, matched on this run's stamp. writeSafetyDump returned one path for an app with two databases; a rollback on that would restore one and leave the other half-written. When the rollback also fails the app is HELD STOPPED (operator ruling): a running app on a half-written database lets the customer make the damage permanent. Every start path refuses it - customer button, appstop Recover, boot sweep - via the shared driveStartGate, checked ABOVE its driveless early return because these apps have no drive. The marker is ended so nothing auto-restarts it. The row goes red. Cleared with --clear-restore-hold, an operator CLI route. --single-transaction is a belt on Postgres only; MariaDB DDL is not transactional and that is why the rollback is the fix. R-381: the engine's stderr stops reaching the customer (615 bytes on MariaDB, its middle rows out of their own database) and starts reaching the operator log, which never had it. R-382: the summary log prints the volume count it already held. Undo copies resolve to their own app, are marked IsUndo, and are capped at 3 per app, pruned from the capture side. The reported render-as-an-app symptom did NOT reproduce - the live page was read first and had zero occurrences. Tests 1468 -> 1483. Eight red-proofs; ONE PASSED and is reported: the R-381 behavioural test injected below ImportDump. A guard at that layer now convicts. |
||
|
admin
|
08eb1a6e3a |
R-356: the off-site restore refused every app that has no data drive
gates / gates (push) Failing after 12s
ReconstituteFromOffsite and PlaceOffsiteRestore both resolved the restore destination with the RAW HDD_PATH and read an empty answer as "the app is not installed". For 40 of the 53 catalogue apps that answer is correctly empty and permanent, so both actions refused forever for a running, healthy app — and told the customer to reinstall it "in the same place", which those apps never offer. Separate the two questions. "Installed?" is asked of ListDeployedStacks via a new Manager.isStackDeployed that fails CLOSED on a nil provider. "Where?" is answered by GetAppDrivePath — the same resolver CaptureRecoveryUnit wrote the snapshot with, so the restore aims at the place the backup came from. The 13 drive apps are unchanged: own drive, mismatch check, ack still required. A third refusal, with its own sentence, covers installed-but-no-resolvable-root. Fixtures that marked an app "installed" by giving it an HDD path now state deployment as its own fact. No assertion weakened. |
||
|
admin
|
5ce3a44645 |
v0.218.0: attribute a DB container by its compose project, and replay volumes on the off-site restore
gates / gates (push) Successful in 11s
R-355 (first, because it is the only one where data can be lost for good). paperless-ngx's PostgreSQL was dumped into backups/primary/paperless/db-dumps/ — a directory for a stack that does not exist, on the system drive — while the app's own unit recorded db_dumps: null. The same misattribution reached writeSafetyDump, so a destructive restore of that app took NO undo copy and the fail-closed refusal was never reached. Fixed by reading the compose project label, which is the stack name by construction (compose runs with cmd.Dir set to the stack dir and no -p). The old derivation stays as the fallback and an unresolvable attribution is now loud. Catalogue sweep, proven able to convict: one affected app of 53. The fix is in the controller, not the catalogue. R-354. ReconstituteFromOffsite skipped every unit placement and the volume archives live inside the unit, so the off-site restore had no volume leg at all — proven live with planted files: calibre-web's 1,422,848-byte config archive was in the unit, the snapshot and the checking folder, and the restore reported success without it. For the 40 of 53 apps that declare no data drive that archive is the whole dataset. restoreDockerVolumesFrom is the local path's own replay with an explicit directory: ONE implementation, two callers. Volumes replay before the database and inside the stopped window. VolumesReplayed reaches the message. The comment beside the skip was half false and is corrected; the half that still holds — the live unit is the local path's source — is named, and scenario D fingerprints the whole live unit across the operation. Seven red-proofs, each asserted applied and reverted. Two found defects in the tests, not the code: scenario D passed with the unit guard removed because the fingerprint had been narrowed and was blind to the unit root. |
||
|
admin
|
a5d90ff801 |
R-241 part 3: abandoning starts a 14-day countdown that ends the question
Until now "set aside" renamed the remote store and touched neither the escrow
nor the key, so the hub went on holding a sealed package for a key the box no
longer used. Shape (c) compares those two, finds them different, and offers
recovery - correctly, and for ever. A customer who had already said "I do not
want the old data" would be asked again at every login.
The operator's ruling is that the answer is NOT a "they decided" flag: fix the
state, do not remember that it is wrong. So the decision starts a countdown,
at the end of which the set-aside store and the sealed package that protects
it are removed TOGETHER. Afterwards shape (c) has nothing to compare and the
offer falls silent on its own - because the state is right, not because
something remembers it once was not.
THE GRACE IS REAL. The recovery offer stays reachable for the whole 14 days;
that is the change-of-mind path, and a grace in which recovery is impossible
would be decorative.
BOTH HALVES OR NEITHER. Removing only the store leaves a package that opens
nothing; removing only the package leaves ciphertext nobody can ever decrypt.
The two cannot be atomic across two machines, so it is a two-phase commit:
delete the store, record a durable marker, and keep DECLARING
offsite.abandon_purge_requested until the hub's ACK stops reporting a
superseded package. A crash between the halves re-declares on the next sweep;
it never leaves the pair half-removed and silent.
HUB HALF - SEC 8.2 ANSWERED: yes, the hub was needed, and only for this.
store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is
ever undone, and it never touches host_escrow (the package covering the key
the box uses now). The handler acts on the DECLARATION, never an inference,
and is placed immediately BEFORE the ACK is built - so
GetEscrowStatusForCustomer reads the effect and the SAME response closes the
box's two-phase commit. No second round-trip and no window where the box
thinks it is still owed. felhom-agent was NOT touched.
The countdown starts in ResetOrphanedRepo, NOT in the shared helper: the
helper is also the unclaimed auto-reset path, where nobody decided anything,
and an as-delivered box tidying a stranger's leftover store must not get a
customer's deletion clock. Pinned by a test.
Cancellation is wired into the recovery unlock, BEFORE the tier-up and the
listing - those can fail, and a countdown surviving a successful unlock
because a later step errored would delete the history the customer just
proved they can open.
The sweep is a Daily job at 05:10, not on the backup leg: it must run on a box
whose tier is not configured for runs. Quiet by construction on every box with
no countdown, and that silence is asserted.
Tests (all clock-injected; SEC 7.4 forbids shortening a live timer):
Scenario E (aside + package kept + countdown + offer still reachable, and
NOTHING deleted), Scenario F (both halves, the declaration repeating, the
close-out), Scenario G (cancel, path still nameable, no later deletion),
plus: not closed out while the package remains, a transport failure leaves the
countdown due and retrying, the no-op sweep issues zero remote commands, and
the unclaimed auto-reset starts no countdown.
RED-PROOFS, each with the mutation confirmed present in the file first:
F1) store deletion skipped -> Scenario F FAILS (no rm issued)
F2) declaration dropped from the report -> Scenario F FAILS (the hub is
never asked; the package would outlive the store for ever)
G) CancelAbandon made a no-op -> Scenario G FAILS (uncancellable countdown)
Green: controller and hub both build, vet and test clean; controller gates OK.
NOTHING WAS DELETED ANYWHERE - the terminal step has only ever run against
in-test fakes.
|
||
|
admin
|
c6b69d888e |
v0.205.0 — a run that skipped an app the customer selected is not successful (R-234)
gates / gates (push) Successful in 21s
THE VERDICT. The R-203 block already said "a warning beside a success is read as a success" and applied it to ONE of the two shapes it describes: an app missing a declared mandatory FOLDER made the run incomplete, while an app skipped ENTIRELY still reported ok. Both do now. Which skips count, decided by measurement: selected+deployed with no recovery unit YES; selected but NOT deployed no (named, with what to do — a box left amber by an app somebody removed is a status nobody reads); disconnected/decommissioned drive no (own signal); nothing selected no. LastSuccess and SnapshotCount still record what WAS captured. THE FILED MECHANISM WAS NOT THE MEASURED CAUSE, and saying so is the point. §3 stated that toggling an app on leaves it without a bundle so the first run skips it. Measured on demo-hp: the run's own pre-dump phase calls captureAllRecoveryUnits for every DEPLOYED stack, through admitApp, before the push — a unit moved aside was RECREATED and the run reported ok. That state does not survive a run. What actually produced the 2026-08-06 sequence: the manual run was dropped by the single-flight while an earlier run was still going. runOffboxBackup returned nil, the handler had already answered "A tavoli mentes elindult", and the card then showed the PREVIOUS run's green verdict — read as covering the app just selected. The decision is now taken synchronously in the handler and a dropped request says so. The nightly path still returns nil on purpose: nobody asked, and it retries. §7.3 measured before deciding: CaptureRecoveryUnit writes a few KB of compose + manifest, only ENUMERATES dumps rather than creating them, is idempotent and does NOT stop the app — and already runs inside the off-site run. So there is no wait to remove for a deployed app and NOTHING was built. 28 packages ok, 9/9 gates. Four red-proofs, each asserted to have applied. Fixture note: the shared provider's ListDeployedStacks returned nil, so Scenario A first passed for the wrong reason; fixed with an opt-in deployed set that defaults to nil. |
||
|
admin
|
58c703bd44 |
R-203 Part 2: a run that missed a MANDATORY directory is not a successful run (v0.197.0)
gates / gates (push) Successful in 8s
The gap was already detected and warned about, in Hungarian, naming the app and the folders -- that warning is what stopped the R-201 drill. The defect was that the run still reported `ok` beside it, and a warning standing beside a success is read as a success. last_status gains "incomplete": minted, because "ok" | "error" | "running" had nothing meaning "it ran, and this app is not fully protected". NOT "error" -- the rest of the run worked and what was captured is real, so SnapshotCount and the LastSuccess anchor still record it. Half a backup is not no backup. The gaps are now recorded STRUCTURALLY (offboxRunResult.mandatoryGaps), not only as prose, so the verdict has something to act on. It reaches the operator through the EXISTING per-run digest (backup_run_failures) rather than a new event type -- a new type is a two-repo change and the hub drops anything outside allowedEventTypes. The stat-filter gains the ClassMandatory check Tier 2 already had. It is a NO-OP today (TierOffsite admits mandatory only), so no customer-visible warning disappears -- demonstrated by widening the tier filter alone and watching the check hold the line. ANTICIPATED: calibre-web on demo-hp has exactly this gap, so its off-site status becomes incomplete the moment this ships. That is correct and is the point. Red-proofs: my first Scenario-C proof PASSED because the test only reached offboxCaptureSet while the mutation lives in runOffboxInternal -- a mutation the test cannot observe is not a red-proof, and the fix was the test. The run-level test now fails under both mutations (unreachable gap recording; unconditional ok). |
||
|
admin
|
73efb091d9 |
R-203: the app and its backup look in the same directory — one resolver, every caller
gates / gates (push) Successful in 9s
appbackup's path helpers take a NAMESPACE ROOT. Five call sites passed a bare DRIVE path.
On an enrolled drive the two coincide, so nothing showed; on the system-data fallback they
differ by exactly the felhom-data segment, and the app then bound a directory the off-site
capture set never looked at -- while the run reported ok. Measured live on demo-hp: the app
wrote to /mnt/sys_drive/userdata/media/books, the capture set looked for
/mnt/sys_drive/felhom-data/userdata/media/books.
THE RULE NOW HAS ONE EXPRESSION. appbackup.NamespaceRootFor / IsEnrolledDrive encode the
drive-kind comparison; backup.Manager.namespaceRoot and stacks.Manager.inGuest delegate to
it. There were already TWO copies and they differed -- the backup package's compared without
filepath.Clean, the stacks package's with it, so a trailing slash from config would have
flipped the mode in one and not the other.
Sites routed through it:
- stacks/deploy.go withPathVars -> ${USERDATA_PATH} (the live defect)
- appexport/fabplan.go + export.go (via a new provider method)
- web/handlers.go FileBrowser mounts (latent: the system drive is
deliberately never a registered StoragePath, so this is the identity today)
ComputeFabBuckets now receives the namespace root, which is what ComputeCaptureSet has always
received -- so the export's classified paths and the backup's capture set describe the same
directories by construction instead of by coincidence.
Tests are table-driven over BOTH drive kinds, because this survived by being invisible on the
kind that already worked. Red-proofs observed: restoring the bare-path call fails the
system-drive row with the two paths differing by /felhom-data; inverting the drive-kind
comparison fails every enrolled row.
|
||
|
admin
|
88897a224e |
v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)
gates / gates (push) Successful in 8s
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed events reached the hub and TWO operator emails went out. The hub's operator cooldown key is customerID:eventType(+tier) and that event carries `app` but no `tier`, so the key held no app identifier: the first refused app took the hour's slot and every other app's failure was discarded BEFORE anything was written down, leaving no row on any channel. The obvious fix — put `app` in the key — was ruled against: on a full disk it produces one email per app, the volume problem wearing the correctness problem's clothes. internal/backup/runsummary.go: a per-run collector with exactly admissionSet's lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the end and only when something failed. A clean run emits nothing. The per-app event stays and becomes the RECORD — the hub routes it record-only, stored and logged every time, never competing for an email slot. The record and the notification are now different things. Deliberate skips (disconnected, decommissioned) are excluded: they have their own alert, and a nightly email about an unplugged drive is one the operator learns to ignore. A manual run always reports: the digest carries a unique run_id the cooldown cannot collapse. Someone pressing the button is actively trying to get a backup. THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a capture failure found between runs would be recorded and never notified — a new silence introduced while closing one. That path emits a digest with NO run_id, so the ordinary 1-hour cooldown caps it exactly as before while the mail now lists every failing app instead of whichever was first. A refusal is recorded ONCE, where the verdict is taken, not at the three legs that consult it — R-181's contract is one verdict per app per run. Noting it per leg listed one refused app three times and produced "2 of 1 apps failed". Found by the digest's own test, not in review. Silence is safe because the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail this box sends (monitor/deadline.go:396,417). Confirmed, not assumed. 7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first attempt — the AST test walked the backup package and not main.go; the test was fixed and the mutation re-run rather than the pass recorded. |
||
|
admin
|
fef07c3923 |
v0.193.0 — the reserve guards the write that fills the disk, and its promise is true (R-181)
gates / gates (push) Successful in 9s
B2's capture floor (v0.192.0) was consulted in exactly ONE place — captureAllRecoveryUnits, which writes a few KB. The two legs that write the BULK into the same backups/primary/<app> tree, the DB dump and the volume dump, ran FIRST and unguarded. Measured live on demo-hp 2026-08-03 06:40:03: opengist's volume dump wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then refused the cheap write it had already lost the argument to. Its refusal message claimed "the previous unit is untouched" — measured false: that app's tar had gone 182,272 B -> 2,147,666,432 B under a stale manifest. Sixth entry in CLAUDE.md's table of shipped guarantees the code did not provide. Fix: ONE admission verdict per app per run (internal/backup/admission.go), taken before that app's FIRST write and covering all three legs — they write under one per-app root, which is why one verdict can honestly cover them. - Lazy, at the app's first write, NOT once at run start: app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. - Remembered for the run, never re-decided between an app's own legs — that is the split this closes. Reset per run. - Placed ahead of DumpAppVolumesSafe, which stops the stack as its first act, so a refused app is never bounced. After the volume-less check, which has no write. - Exactly one operator alert per refused app per run. - Leg order unchanged: volume dumps still precede the capture. The floor is now SIZE-AWARE: it asks whether THIS app's write would cross the reserve, not only whether the filesystem is already below it — which is how an app was admitted at 96% and then allowed to write 2 GB. Estimate = the app's previous .sql + .tar on disk. No history -> headroom-only, deliberately, and the alert says so. A container-based du per volume was MEASURED and rejected: 66 timed runs on demo-hp guest 9201, median ~355 ms/volume (341-404) on volumes holding tens of KB — container start-up, not the walk. Decisive on top: docker run needs the writable layer, so it can fail under exactly the pressure the reserve handles. The message was NOT weakened; the behaviour was moved so the wording became true. It now also names which term bound. Every claim is checked against a sha256 fingerprint of the tree it describes, never against the log line. Still refuses and never deletes: nothing here is generational. 11 new tests through the production functions. The DB leg cannot run without Docker, so its gate is pinned by an AST walk of backup.go asserting admitApp precedes DumpOne (strings.Contains is insufficient — a commented-out call still contains the string). 4 red-proofs demonstrated failing then restored. |
||
|
admin
|
4be6467b50 |
v0.192.0 — the capture floor replaces the bulkhead (R-165, decision B2)
gates / gates (push) Successful in 8s
Ships BEFORE the disk-layout merge it exists for, and is harmless on a box that never gets it. The mp1 partition was a BULKHEAD as well as a ceiling: it kept a runaway capture from filling the space the container runtime needs, because /var/lib/docker was a different filesystem. After the merge it is the same one, and a full Docker data-root is a stopped box. The floor sits in captureAllRecoveryUnits, checked BEFORE anything is written: below the reserve, that ONE app's capture is refused, its previous unit is left byte-identical, the R-158 alert fires with the space figures, and the loop continues. Two terms whichever binds first (97% used / 1 GiB free) in fillwatch's shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned before a refusal can happen — a floor that fires before its own warning is a silent failure wearing a threshold. Headroom, never unit size: a per-unit cap would be R-163 rebuilt inside one volume. Refuses, never deletes: nothing here is generational, so pruning could only destroy a different app's only local copy; pruneStalePrimaryDirs is an orphan sweep, not retention, and must not be repurposed. Tests 1184 -> 1191. One fixture strengthened mid-red-proof: the "old 20 G ceiling is gone" test sat at exactly 20 GB and survived a literal UsedGB > 20 cap — hollow. Now 120 GB, and the mutation fails it. |
||
|
admin
|
cf48214f6c |
v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored. |
||
|
admin
|
dbcb306fcf |
v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b)
gates / gates (push) Successful in 8s
The box stops inferring the customer's intent from a container count and reads
what they actually asked for.
Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).
ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.
Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).
Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.
No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
|
||
|
admin
|
9f436c8a3b |
v0.150.0 — green gate restored + the export link stops leaking the CSRF token
F7/R-53: app_export.html built the app's public URL as '<sub>.{{$.CSRFToken}}',
so the "Megnyitás" link was wrong for every app with a subdomain and a session
CSRF token was written into a URL. Template now uses {{$.Domain}}, and
exportPageHandler supplies the key — it builds its own data map instead of
going through baseData, which is where every other page gets it. The page's
real CSRF path (csrfH() reading the meta tag) is correct and untouched.
The 7 red internal/backup tests are green again, with no behaviour change.
TestTier2V2_* / TestSharesTier2* all failed for one environmental reason:
Tier-2's off-drive guard asks system.SamePhysicalDevice (st_dev equality)
whether a target is really a second disk, and every t.TempDir() here shares one
filesystem — so the guard correctly refused the fixture's "two drives" and the
tests never reached their subject ("nincs másik fizikai meghajtó").
Seam in the package's existing style: a nil-defaulted Manager.samePhysicalDevice
field + sameDevice wrapper, seven call sites routed through it. Nil resolves to
system.SamePhysicalDevice, so production is byte-for-byte unchanged; only the two
fixtures inject a fake modelling one drive per directory subtree. No assertion
weakened, nothing skipped/renamed/deleted; all 7 mutation-proved.
Also: the ssh->pct-exec ASCII-grep and heredoc-credential traps are now in
CLAUDE.md's live-validation section.
|
||
|
admin
|
062357f778 |
v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
|
||
|
admin
|
b5d78d1e0f |
v0.147.0 — feedback slice 1: pressing a button says something
The systemic complaint, twice in one evening: you press a button and nothing happens. No progress, no ETA, no named result. Three worst offenders, fixed on the two patterns already here (deploy 3-step panel, storage-init status poll). No new framework — that is a ROADMAP item; three targeted cards ship tonight. 4a — a verification restore names its result. The flash said the app had been restored "to a verification folder on the drive"; which folder, on which drive, was invisible, so the customer could not go and look at what they had just asked for. Full path now. The restore page gained a listing of existing verification copies (app, size, date, path) — nothing anywhere showed these, so they piled up and the only way to find them was SSH — each with a double-confirmed delete. That delete is the only one this release adds, so it names a STACK, never a path: the Manager resolves the name inside a backups/offsite-restore root it computed itself and refuses anything landing outside. Red-proofed — neutralise the name guard and stack:"" resolves to the offsite-restore ROOT and takes every copy with it. Refusals are asserted as non-effects. 4b — Megosztás enable shows what it is waiting for. Enabling ran ReconcileSamba synchronously inside the POST handler; on a golden without felhom-samba baked that is compose pulling ~100MB, i.e. minutes of an apparently-hung form post followed by "Beállítás mentve." whether or not anything came up. Detached + polled now, distinguishing "képfájl letöltése" from "indítás" — decided BEFORE the work starts, since afterwards the image is always present. Success is probed, not inferred (compose up -d exits 0 on a crash-loop). The password form starts the same job: with UserSet false reconcile deploys nothing, so on a fresh box that is where the pull actually happens. 4c — "Távoli mentés most" streams real progress. restic was already reporting bytes and percent; the runner seam used CombinedOutput() and discarded them. The manual run now passes --json and scans stdout line-by-line: total bytes, percent, current app. Manual only — the nightly stays silent, pinned by a test that fails if it ever passes --json. The poll now arms unconditionally, closing a race the manual trigger always ran: the redirect rendered before the goroutine wrote LastStatus=running, so the poll never armed and the page sat static during the very run just started. Red-proofed twice. Also closes the golden/controller infra-image drift at the source: infra.Images() derives from the existing pins and --print-infra-images exposes it, so the golden bake can stop carrying its own copy. That copy had already drifted — felhom-samba was never added, so the golden baked 3 of 4, which is why enabling Megosztás pulled at runtime in the first place. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE |
||
|
admin
|
900c870212 |
feat(shares): R-7b Parts 4-6 — shares restore, samba liveness, UI truth-up
Part 4 — restore: RestoreSharesScratch + PlaceSharesRestore as SIBLINGS of the
per-app scratch/place pair. Files merged missing-only (never overwriting), each
destination PREFIX-ASSERTED against registered LIVE storage roots; definitions
merged with existing-wins; ReconcileSamba via a seam (backup must not import
stacks); credential restored best-effort into the samba named volume.
New routes POST /backup/shares/{restore,place} + a restore-page entry that renders
'Megosztasok', never the raw reserved key.
Also adds scratchJoin: reconstructing an absolute captured path under a scratch
must strip the volume name rather than rely on filepath.Join.
Part 5 — liveness: EffectiveProtected gains a settings-backed dynamic extra so the
samba CONTAINER (not the stack name — they differ) is watched exactly while sharing
is on. FINDING: the issue -> health 'fail' -> existing health_critical event ->
alert -> Hungarian degradation e-mail path needs NO further change, and introduces
no new event type, so the allowlist gotcha does not apply.
Part 6 — UI: per-tier backup status lines on the Megosztas page (amber only on
deviation). Verified the two warning-prose sites (offbox_capture/tier2_capture)
only ever receive per-app stack names, so no mapping is needed there.
RED-PROOFS RUN AND REVERTED (both fired):
4. prefix-assert removed -> place-guard traversal test FAILS
5. dynamic samba extra removed -> Scenario E enabled-case FAILS
|
||
|
admin
|
c81df55dcb |
feat(shares): R-7b Parts 1-2 — shares payload builder + tier-2 shares job (Model B')
Sibling shares source for the local cross-drive tier. Reuses the tier2Mirror seam, selectTier2TargetFrom (narrow source-drive seam extracted from selectTier2Target), tier2ReconcileRoots (pure extraction), tier2SafeRemove, the marker-LAST discipline and the recordTier2* helpers. Per-app paths are untouched. - shares_payload.go: deterministic _shares-manifest.json + best-effort passdb capture - tier2_shares.go: per-source-drive legs -> cross-drive target, payload, marker LAST - infra.SambaContainerName/SambaPassdbVolume/Mount: single source of truth for the container identity (renderer, stacks execs, backup execs, monitor all read it) - RESERVED-NAME finding: ValidateSMBShareName did NOT exclude a leading underscore, so "_shares" was an accepted share name. Now refused; RunAllTier2 additionally skips a "_shares" stack loudly as defense in depth. - fix: shareSourceDrive returned a slash-normalised path, which made the target selector's source-drive equality check miss (a group could target its own drive) |
||
|
admin
|
596505ed64 |
v0.142.0: offsite repo continuity — orphaned-repo guard (A) + run-status auto-refresh (C)
- Part A: classify restic cat-config failure (wrong-password=orphaned vs no-repo vs other); ORPHANED state + Hungarian card + offbox_repo_orphaned/reset events (once, not nightly); reset = move-aside (never delete) + init, unclaimed auto / claimed confirm. Red-proofs TestOffbox_OrphanDetection_* + ConfirmedReset. - Part C: GET /backup/offbox/status + poll on backups_remote → flips Fut→Rendben/Hiba without manual reload. |
||
|
admin
|
3603d1fc7f |
Tier-2 engine rework: class-driven legs, v2 layout, NAS-target exclusion (Task 3b, v0.135.0)
tier2_capture.go: classified apps get TierSecondary per-bind legs (paperless copy shrinks — export
drops); legacy apps keep the byte-identical resolver set. v2 relpath-mirroring layout
(backups/secondary/<stack>/{marker LAST, recovery-unit/, hdd/<rel>/, userdata/<rel>/}); N>1 native
(errTier2MultiDir/tier2AppDataName deleted). Migration=delete-and-rebuild + reconcile; all RemoveAll
via tier2SafeRemove (refuses outside backups/secondary/). SSD=state-only tier. selectTier2Target
never picks network storage (pinned+auto, F-6C-1). Restore reads v2 behind a marker gate.
Part 0: offbox_enlarge_blocked is a persisted one-time Load seed (opt-out sticks), not a getter
append. Part 0.5: offsite restore scratch prefers a local (non-network) path.
Full v2 test suite + all 10 §10 red-proofs verified. Destructive writes bounded to backups/secondary/.
|
||
|
admin
|
2d20859858 |
Offsite tier policy engine: mandatory userdata, raw-data quota, restore rework (Task 3a, v0.134.0)
Each toggled app's offsite push = one multi-path restic snapshot (recovery unit + TierOffsite mandatory userdata via ComputeCaptureSet); legacy/undeployed stay unit-only. Loud capture gaps (SP-3.4: restic 0.14.0 silently skips missing paths). Quota = stats --mode raw-data (SP-1; displayed size drops once). Pre-push enlargement gate blocks the userdata enlargement over-quota (unit-only push continues; EnlargedBlocked; edge-triggered notify). forget --group-by host,tags on both sites (SP-2). Restore reworked: scratch off the rootfs + headroom gate (F-A1), unit-only default via --include, size-first full, place-to-live missing-only merge (never --delete). UI: unit/full-two-step/place actions + per-app blocked note; route POST /backup/offbox/place. HUB FLAG: offbox_enlarge_blocked event needs hub allowlist for push delivery. +13 tests; all 10 §10 red-proofs verified. No tier-2/.fab/hub/agent changes. |
||
|
admin
|
68f0e0cf5c |
F-S2 + F-S3: compose-derived appdata dir resolution (v0.131.0)
The controller assumed an app's HDD appdata dir is always appdata/<stackName>.
paperless-ngx writes appdata/paperless (stack paperless-ngx), so every consumer
keying by stack name silently missed it via a stat-and-skip. One canonical
resolver appbackup.AppDataDirNames derives the real dir name(s) from the app's
compose ${HDD_PATH} binds; all consumers use it.
- F-S2 (tier-2): RunTier2 mirrors the resolved appdata/<name> (paperless docs
got NO tier-2 copy before). Tier2Info size + RestoreTier2Files live dir use it.
WARN when a declared appdata dir is absent. New tier2Mirror seam.
- F-S3 (migrate, NEW): all six per-app appdata legs (collision/size/copy/verify/
cleanup/skip-set) now loop resolved names. scope="app" migration of paperless
previously copied nothing and left an empty media dir (scope="all" was saved by
the merge walk). WARN on missing declared dir in the copy leg.
- Multi-dir (N>1) refusal: tier-2 backup/info/restore refuse loudly (Hungarian);
migrate supports N. No catalog app hits it today; lifted by Task 3.
- Display: storage page sums resolved dirs.
- Truth repair: the v0.130.0 "tier-2 copies the namespace wholesale" claim is
false; corrected in CHANGELOG + main.go export-adapter comment.
+9 tests; red-proofs RP-1..RP-5 all confirmed. Controller-only, no agent/hub
coupling. Task 1 of the backup-classification-redesign arc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
|
||
|
admin
|
40b53047f1 |
controller: atomicPromoteTar syncs via an O_RDWR handle — read-only fsync is refused on Windows, which kept the dev-box green gate permanently red (Linux behavior unchanged)
Claude-Session: https://claude.ai/code/session_01GzammAMzsJTgpQHqxwM2bC |
||
|
admin
|
68b3a3932e |
controller: F7 atomic volume dumps + F6 no-single-copy + F5 stale-primary sweep (WIP, pre-build)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf |
||
|
admin
|
cb9e992c7a |
v0.110.0: offbox stale-lock self-heal (campaign C2) + crash-truthful status (C1)
resticStep escalates a restic lock error to `unlock --remove-all` + one retry (safe: single-writer repo — sub-account isolation + single-flight mutex); plain `unlock` is stale-only and can't clear a crash lock across a container-hostname change. Pre-run stale unlock hygiene on run+restore. C1: NewManager flips a persisted LastStatus=running to a truthful error. Both red-proofed (A reproduces the exact campaign backup failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 |
||
|
admin
|
c529a455af |
feat(backup): async restore family — no proxy-timeout error page on a succeeding restore (v0.102.0)
Re-adjudicates F4: /backup/restore, /backup/tier2/restore, /backup/offbox/restore blocked the HTTP request until completion, so through cloudflared's 100s cap a customer got an error page while the restore succeeded (offbox worse — bounded on r.Context(), canceling the SFTP restore mid-flight). Convert all three to the offboxRun async shape: fast-path IsRunning refuse, background goroutine (offbox ctx off r.Context() -> Background+30m), instant redirect. Add mutex- guarded op-status (opstatus.go) + GET /api/backup/restore-status + a 3s-polling backups.html banner (neutral running, red on failure). Restore single-flight unchanged. Tests + red-proof (sync handler blocks indefinitely vs <500ms async). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 |
||
|
admin
|
30b11100e0 |
feat(backup): C2 Part 1 — RestoreTier2Files: in-place, additive-only class-C file restore engine
Closes the engine half of drill finding F2: user files under appdata/<stack> had no customer recovery path (operator copy-back only). - RestoreTier2Files(stack): single-flight with backup/restore; ALL refusals before any stop (no Tier-2 record / LastRun empty / copy dir absent → "nincs másodlagos fájlmásolat"; Tier-2 drive disconnected; live drive disconnected/decommissioned — Hungarian, flash-ready); source is the RECORDED CrossDriveBackup.DestinationPath (never a fresh selectTier2Target); stop → copy → start → waitForHealthy; copy/restart errors surface (F17). - rsyncRestoreMissing: rsyncMirror's exec shape with the OPPOSITE-direction flags: -a --ignore-existing --itemize-changes — existing live files are never overwritten, nothing is ever deleted (the --delete trap this task exists to avoid). Count = ">f" itemize lines (pure countRestoredFiles). - restoreFilesCopier seam so orchestration tests never shell out; the one FS-level test of the real rsync is LookPath-guarded (runs on the Linux build server + live validation). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 |
||
|
admin
|
a52851e79e |
fix(backup): O4 — generate a replacement for unrecoverable resettable secrets on restore
The proceed-path for a missing RESETTABLE secret redeployed the app with the secret blank (compose "Defaulting to a blank string" → exit 1, live-hit in the 2026-07-04 drill Phase 5). Now the restore generates a fresh credential instead: - stacks.Manager.GenerateSecretForField: replacement value from the field's catalog generate spec via the deploy flow's generateValue (no logic copied); refuses data-keys (defense-in-depth), spec-less and non-secret fields. - backup.Manager.SetSecretGenerator seam (wired in main.go), consulted in RestoreFromRecoveryUnit AFTER the untouched fail-closed gate, for missing names NOT in DataKeyEnvVars. The generated value rides fullEnv into RecreateStackFromUnit → RedeployFromEnv → SaveAppConfig, so it persists encrypted in the guest app.yaml and round-trips on the next backup/restore (no second write path). reconcileRestoreSecrets stays pure and untouched. - WARNs now discriminate: "generated replacement for X (credential was reset)" vs "X unrecoverable and has no generator — app may fail to start". Values are never logged (asserted in test). - Residual case (documented, not pretended away): if a restored volume tar carries the OLD internal credential hash, the app may still fail auth until a manual in-DB reset — generation fully fixes only the fresh-init case. Companion red-proof: pre-fix behaviour (generation skipped) fails TestRestoreGeneratesMissingResettableSecret on the non-empty DB_PASSWORD assertion (verified, reverted). Data-key gate proven unreachable by generation in TestRestoreGenerationNeverReachesDataKeys. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 |
||
|
admin
|
73378a812c |
fix(backup): F3 — wire named-volume dumps into the app-data backup run
DumpAppVolumesSafe had NO production caller: no trigger ever produced
volume-dumps/, so named-volume app data (e.g. nextcloud's html volume) was never
captured into the recovery unit and the granular restore silently restored
nothing for class-B data (drill finding F3).
- runVolumeDumps: per-stack loop in runDBDumpsInternal, BEFORE
captureAllRecoveryUnits (so manifests enumerate the fresh tars). Gate order is
load-bearing: protected-stack and volume-check gates precede DumpAppVolumesSafe
(which stops the stack before its own check — unconditional calls would bounce
every volume-less app nightly). Disconnected/decommissioned drives skip with
the same summary style as the DB loop.
- No silent partials: a per-stack failure lands as a FAIL summary entry, flips
Success, and fails the run ("some backup steps failed: ..."), without aborting
the other stacks.
- Zero-DB early return removed: volume-bearing apps without a database still get
their class-B dump + unit refresh.
- dumpVolumesSafe seam (same style as the F17 discoverDBs/importDBDump seams) so
the gating is unit-tested without Docker. Companion red-proof: neutering the
volume gate fails TestRunVolumeDumps_GatesPrecedeDump (dump fired for the
volume-less stack) and _VolumelessNeverStopped (verified, reverted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
|
||
|
admin
|
2a7deadc93 |
controller v0.93.0: NAS Part B off-box backup target (restic-over-SFTP)
Encrypted restic repo over SFTP for the app-data tier (the off-site 3-2-1 leg). A dead NAS fails fast via -oConnectTimeout (spike Q8), never hangs the runner; secrets are 0600 files (ride DR via PBS whole-CT); init-if-absent, retention forget --prune, restore, single-flight, per-app toggle + UI. restic re-added to the image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs |
||
|
admin
|
b57150e3ab |
B1: data-migration engine (MigrateAll + MigrateApp) + backup mutual-exclusion
internal/stacks/migrate.go: crash-safe, resumable namespace migration over the controller's /mnt RW mount. Two entry points (whole-namespace + per-app) share one journaled pipeline: validate -> stop -> copy (rsync -a --checksum, additive; conflict- merge walk for non-app content) -> verify -> flip+redeploy (RedeployFromEnv) -> cleanup. CLEANUP (the only destructive step) is gated on all units verified AND all apps redeployed. Single-flight; mutual exclusion with the backup orchestrator (Change 3). Non-hollow tests incl. mutation-proven collision + cleanup-gate companions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
admin
|
f8afe5c055 |
M18: cache DB-dump validation; skip re-validate on unchanged dumps
ListDumpFiles ran ValidateDump (line-by-line scan) for every dump on every ~5-min RefreshCache cycle — wasted I/O+CPU on large customer dumps. ListDumpFiles now takes an optional cached(name,size,mod) lookup; on a (size+modtime) match it reuses the prior result and skips ValidateDump. settings.DBValidationCache gains Size+ModTime; listAllDumpFiles builds the lookup from the persisted cache and writes back only fresh validations (cache miss), so an unchanged dump triggers neither a re-validation nor a settings.json write each cycle. nil cached = legacy validate-always (back-compat). Tests: cache-hit skips validate (sentinel), cache-miss validates, nil validates. |
||
|
admin
|
6bab68b132 |
M19: deriveStackName cross-references deployed stacks (fix DB-container misattribution)
deriveStackName pure-suffix-stripped on '-' (postgres/db/mariadb/.../cache), so a stack whose slug ENDS in a role token (e.g. 'my-cache') was misattributed (stripped to 'my') — filing its DB dump under the wrong/nonexistent stack. Now threads the set of deployed stack names (m.knownStackNames() <- ListDeployedStacks) into DiscoverDatabases and cross-references: candidate suffix-strip if known, else the container name if it IS a known stack, else longest known stack that is a prefix (handles <stack>_postgres / <stack>-1), else legacy strip. nil/empty known = legacy behaviour (appexport passes nil). Table test incl. the my-cache case (fails pre-fix). |
||
|
admin
|
0b9450e356 |
F17: per-app restore now replays the captured .sql DB dump (.sql wins)
The restore paths (RestoreFromRecoveryUnit + the RestoreApp fallback) repopulated Docker volume tars but NEVER replayed the captured <stack>-<dbtype>.sql dump, so DB-resident data (e.g. rows in a DB whose data dir is a bind mount) did not come back — the romm marker round-trip in the audit lost the row. New appbackup.ImportDump (read-side counterpart to DumpOne) replays a .sql/.sql.gz into the running DB using the live container's OWN discovered credentials (no env threading; reuses DiscoveredDB + getMariaDBPassword). backup.reimportDBDumps orchestrates it AFTER volume restore + stack bring-up, so the logical dump WINS over any volume-tar copy of the DB (operator-chosen precedence). pg_dump --clean --if-exists and mariadb-dump (default --add-drop-table) make replay idempotent; psql ON_ERROR_STOP=1 surfaces real import errors. Also: volume-restore per-volume failures and DB-import failures now SURFACE (the restore returns an error) instead of a swallowed WARN, so a failed data restore cannot read as success. Tests (restore_db_test.go, injectable discover/import seams): imports when dump+DB present, failure surfaces, no-dump skips discovery, dump-but-no-matching-DB is a non-fatal skip. Live DB round-trip to be validated post-deploy. |