Compare commits

...

3 Commits

Author SHA1 Message Date
admin 4c8f0d2919 R-103: the Tier-2 refusal becomes an action
gates / gates (push) Successful in 12s
An app whose Tier-2 copy holds no file legs but a full recovery-unit mirror - 45 of the 53 catalog
templates - was told to press a button on a DIFFERENT page. Since R-102 the data it is asking for is
restorable from the copy it is looking at.

New POST /backup/tier2/unit-restore and backupTier2UnitRestoreHandler: same guards, same
restoreOpBlocked() refusal (R-351b), same async shape as the file restore beside it, plus a
fail-closed pre-flight so the app is never stopped for a mirror that could not be opened. The
outcome reuses unitRestoreOutcomeMsg and adds which copy overwrote the live data.

The row offers the action where the refusal was, in a danger style, as a SEPARATE button. The two
are not merged: one adds what is missing, the other overwrites. The confirm carries that difference
in words and names the copy's date - and says so differently when that date is only an ATTEMPT
(R-101). It is built from named Go constants rather than assembled inside an HTML attribute, so a
test can assert it verbatim; fmtTimeStr now delegates to a package-level fmtRFC3339Local so the
confirm and the outcome cannot render the same date two ways.

tier2NoCoverageMsg is NARROWED to the case that remains - no legs and no openable unit - and still
names the route that works. tier2UnitNotCoveredMsg is NOT deleted: it is appended where the FILE
restore ran and is still exactly true of it.

Tests C1-C2 and D1-D6 plus four more. Red-proofs: C1 (widen CanRestore to include HasUnit -> the
unit-only cases fail), D6 (drop EndRestoreOp from the handler goroutine -> 'the restore never
published a result').
2026-08-31 11:42:03 +02:00
admin 0f9b796615 R-102: the recovery unit on the second drive becomes a way back
Tier-2 mirrors each app's whole recovery unit to <dest>/backups/secondary/<app>/recovery-unit/ on
every run and has done for months. Nothing read it. In the one failure Tier-2 exists for - the
primary drive is lost, and the primary unit with it - the surviving copy could not be opened by any
action in the product (07-backup-architecture 6.3, 7.2).

Part 1.2: RestoreFromRecoveryUnitAt(stack, unitDir) holds the whole body; RestoreFromRecoveryUnit is
the thin caller naming the primary unit. ONE implementation, two callers. The SOURCE moves; the
DESTINATION does not - live Docker volumes, the live database container, the guest's definition, all
unchanged. The R-47 mutation order, the secret reconciliation with unit-over-guest precedence, the
fail-closed data-key gate and the no-unit fallback with CountsUnknown are untouched.
reimportDBDumpsAtCtx is the bounded-context twin of reimportDBDumpsFrom; the 35-minute bound is now
named once so the two paths cannot drift. The R-354 volume-replay seam is reused rather than a second
one invented, which is what lets the acceptance test assert the volume leg's source directory.

Part 1.3: RestoreTier2Unit resolves the recorded copy, refuses fail-closed unless the mirror carries a
parseable manifest - a directory is not a package - and delegates. The single-writer flag is taken
inside RestoreFromRecoveryUnitAt, not beside it.

Part 2.1: Tier2Coverage gains UnitRestorable and the copy's dates. CanRestore() is NOT widened; it
still answers only 'can the file restore run?'. One predicate answering two questions is R-356, which
refused 40 running apps for months.

Tests: A2-A6 and B1-B5, plus two non-regression guards. The Tier-2 fixtures build their mirror with
the production RunTier2, so the claim is 'the copy Tier-2 writes is the copy this restore reads'.
Red-proofs: A5 (swap volumes/recreate -> fails on the order), B2 (point the reader back at the
primary -> fails with the mirror never reaching the redeploy, and with permission denied once the
primary tree is unreadable).
2026-08-31 11:30:34 +02:00
admin c732006d26 R-102 phase 1.1: split the recovery-unit path helpers, zero behaviour change
Every unit path helper took (nsRoot, stackName) and joined backups/primary/<stack>/... . That
hard-coded 'primary' is the mechanism of R-102: Tier-2 mirrors the whole unit directory to
<dest>/backups/secondary/<stack>/recovery-unit/ every night, and because no reader could NAME a
unit outside backups/primary/, that mirror has been captured for months and read by nothing.

Adds four unit-directory-relative primitives - UnitComposeDir, UnitManifestFile, UnitDBDumpDir,
UnitVolumeDumpDir - each taking the recovery-unit DIRECTORY itself. The four existing
(nsRoot, stackName) helpers become thin wrappers over them and keep their exact signatures and
their exact return values; every current caller compiles untouched.

ONE implementation, two callers - the rule restoreDockerVolumesFrom already follows in this repo.

TestR102_PathWrappersAreByteIdenticalToToday pins the wrappers against hand-written literals (not
re-derived from the helpers under test). Red-proof: UnitComposeDir join changed to 'compose2' ->
the test fails on all three fixtures.
2026-08-31 11:16:02 +02:00
14 changed files with 1735 additions and 36 deletions
+39 -4
View File
@@ -77,25 +77,60 @@ func RecoveryUnitPath(nsRoot, stackName string) string {
return filepath.Join(nsRoot, "backups", "primary", stackName)
}
// UNIT-DIRECTORY-RELATIVE HELPERS (R-102). The four below take THE RECOVERY-UNIT DIRECTORY ITSELF
// and know only the unit's internal layout. The `(nsRoot, stackName)` helpers that follow are thin
// wrappers over them, and every existing caller keeps its exact signature and its exact result.
//
// They exist because the `(nsRoot, stackName)` form resolves through RecoveryUnitPath, which joins
// `backups/primary/<stack>` — a hard-coded `primary` that was the MECHANISM of R-102. Tier-2 mirrors
// the whole unit directory to `<dest>/backups/secondary/<stack>/recovery-unit/`, and because every
// reader of a unit could only name a `primary` path, that mirror was captured nightly for months and
// read by nothing. The unit's INTERNAL layout is identical wherever the directory sits, so the fix is
// to let a reader name the directory rather than re-derive it.
//
// ONE implementation, two callers — the same rule as backup.restoreDockerVolumesFrom. Do not add a
// second copy of a join: a duplicated layout constant is how the two sides drift apart.
// UnitComposeDir returns the compose/config capture dir within a recovery-unit DIRECTORY
// (docker-compose.yml + .felhom.yml + app.yaml carrying the portable secret class).
func UnitComposeDir(unitDir string) string {
return filepath.Join(unitDir, "compose")
}
// UnitManifestFile returns the manifest.json path within a recovery-unit DIRECTORY.
func UnitManifestFile(unitDir string) string {
return filepath.Join(unitDir, "manifest.json")
}
// UnitDBDumpDir returns the DB dump directory within a recovery-unit DIRECTORY.
func UnitDBDumpDir(unitDir string) string {
return filepath.Join(unitDir, "db-dumps")
}
// UnitVolumeDumpDir returns the Docker-volume dump-tar directory within a recovery-unit DIRECTORY.
func UnitVolumeDumpDir(unitDir string) string {
return filepath.Join(unitDir, "volume-dumps")
}
// RecoveryUnitComposePath returns the compose/config capture dir within an app's recovery unit
// (docker-compose.yml + .felhom.yml + secret-stripped app.yaml).
func RecoveryUnitComposePath(nsRoot, stackName string) string {
return filepath.Join(RecoveryUnitPath(nsRoot, stackName), "compose")
return UnitComposeDir(RecoveryUnitPath(nsRoot, stackName))
}
// RecoveryUnitManifestPath returns the manifest.json path within an app's recovery unit.
func RecoveryUnitManifestPath(nsRoot, stackName string) string {
return filepath.Join(RecoveryUnitPath(nsRoot, stackName), "manifest.json")
return UnitManifestFile(RecoveryUnitPath(nsRoot, stackName))
}
// AppDBDumpPath returns the DB dump directory for an app under a felhom-data namespace root.
func AppDBDumpPath(nsRoot, stackName string) string {
return filepath.Join(RecoveryUnitPath(nsRoot, stackName), "db-dumps")
return UnitDBDumpDir(RecoveryUnitPath(nsRoot, stackName))
}
// AppVolumeDumpPath returns the Docker-volume dump-tar directory for an app under a namespace root.
func AppVolumeDumpPath(nsRoot, stackName string) string {
return filepath.Join(RecoveryUnitPath(nsRoot, stackName), "volume-dumps")
return UnitVolumeDumpDir(RecoveryUnitPath(nsRoot, stackName))
}
// AppDataDir returns the app data directory under a felhom-data namespace root. The final segment
@@ -0,0 +1,73 @@
package appbackup
import (
"path/filepath"
"testing"
)
// A1 — TestR102_PathWrappersAreByteIdenticalToToday.
//
// R-102 Part 1.1 split each unit path helper into a unit-directory-relative primitive plus a thin
// `(nsRoot, stackName)` wrapper. The refactor's whole claim is ZERO behaviour change, and the only
// honest way to pin that is to compare each wrapper against the literal string it produced before the
// split — written out here by hand, not re-derived from the helpers under test (a test that called
// UnitComposeDir to compute its own expectation would pass for any consistent pair of wrong joins).
//
// Red-proof (recorded in REPORT.md): change one wrapper's join — e.g. UnitComposeDir to "compose2" —
// and this test fails.
func TestR102_PathWrappersAreByteIdenticalToToday(t *testing.T) {
cases := []struct {
nsRoot string
stack string
}{
{"/mnt/hdd1", "docmost"},
{"/mnt/sys_drive/felhom-data", "immich"},
{"/mnt/felhom-usb", "paperless-ngx"},
}
for _, c := range cases {
unit := c.nsRoot + "/backups/primary/" + c.stack
if got, want := RecoveryUnitPath(c.nsRoot, c.stack), unit; got != want {
t.Errorf("RecoveryUnitPath(%q,%q) = %q, want %q", c.nsRoot, c.stack, got, want)
}
if got, want := RecoveryUnitComposePath(c.nsRoot, c.stack), unit+"/compose"; got != want {
t.Errorf("RecoveryUnitComposePath(%q,%q) = %q, want %q", c.nsRoot, c.stack, got, want)
}
if got, want := RecoveryUnitManifestPath(c.nsRoot, c.stack), unit+"/manifest.json"; got != want {
t.Errorf("RecoveryUnitManifestPath(%q,%q) = %q, want %q", c.nsRoot, c.stack, got, want)
}
if got, want := AppDBDumpPath(c.nsRoot, c.stack), unit+"/db-dumps"; got != want {
t.Errorf("AppDBDumpPath(%q,%q) = %q, want %q", c.nsRoot, c.stack, got, want)
}
if got, want := AppVolumeDumpPath(c.nsRoot, c.stack), unit+"/volume-dumps"; got != want {
t.Errorf("AppVolumeDumpPath(%q,%q) = %q, want %q", c.nsRoot, c.stack, got, want)
}
}
}
// TestR102_UnitHelpersAreDirectoryRelative pins the OTHER half of the contract: the four primitives
// join onto whatever directory they are handed, with no `primary` segment reintroduced. The secondary
// mirror path below is the exact shape Tier-2 writes (tier2.go: `<dest>/backups/secondary/<stack>/
// recovery-unit`), so a regression that re-derived a namespace root would show up here as a `primary`
// appearing in a secondary path.
func TestR102_UnitHelpersAreDirectoryRelative(t *testing.T) {
unit := "/mnt/hdd2/backups/secondary/docmost/recovery-unit"
checks := []struct {
name string
got string
want string
}{
{"UnitComposeDir", UnitComposeDir(unit), unit + "/compose"},
{"UnitManifestFile", UnitManifestFile(unit), unit + "/manifest.json"},
{"UnitDBDumpDir", UnitDBDumpDir(unit), unit + "/db-dumps"},
{"UnitVolumeDumpDir", UnitVolumeDumpDir(unit), unit + "/volume-dumps"},
}
for _, c := range checks {
if c.got != c.want {
t.Errorf("%s(%q) = %q, want %q", c.name, unit, c.got, c.want)
}
if filepath.Base(filepath.Dir(c.got)) == "primary" {
t.Errorf("%s reintroduced a primary segment: %q", c.name, c.got)
}
}
}
@@ -156,6 +156,18 @@ func RecoveryUnitManifestPath(nsRoot, stackName string) string {
return appbackup.RecoveryUnitManifestPath(nsRoot, stackName)
}
// The four UNIT-DIRECTORY-RELATIVE helpers (R-102). They take the recovery-unit DIRECTORY itself, so
// a reader can name a unit that is NOT under backups/primary/ — the Tier-2 mirror at
// <dest>/backups/secondary/<stack>/recovery-unit/ being the whole point.
func UnitComposeDir(unitDir string) string { return appbackup.UnitComposeDir(unitDir) }
func UnitManifestFile(unitDir string) string { return appbackup.UnitManifestFile(unitDir) }
func UnitDBDumpDir(unitDir string) string { return appbackup.UnitDBDumpDir(unitDir) }
func UnitVolumeDumpDir(unitDir string) string { return appbackup.UnitVolumeDumpDir(unitDir) }
func AppDataDir(nsRoot, stackName string) string {
return appbackup.AppDataDir(nsRoot, stackName)
}
@@ -0,0 +1,298 @@
package backup
import (
"context"
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-102 Group B — the Tier-2 route: the mirror on the SECOND DRIVE becomes a way back.
//
// The mirror in every test here is written by the PRODUCTION Tier-2 run (RunTier2 with copyTree
// standing in for rsync, the seam tier2_v2_test.go already uses), not by hand. That matters: the
// claim is "the copy Tier-2 actually writes is the copy this restore reads", and a hand-built
// fixture could only prove that the restore agrees with the test's idea of the layout.
// r102T2 is a fully-wired Tier-2 fixture: a live drive carrying the app's PRIMARY recovery unit, a
// second drive carrying the mirror Tier-2 just wrote, and a restore-side Manager with the Docker and
// DB seams injected so no daemon is touched.
type r102T2 struct {
m *Manager
fake *fakeRecoveryProvider
liveDrive string
destDrive string
destBase string
primaryUnit string
volDirs *[]string
dbPaths *[]string
}
// r102Tier2Fixture captures a unit on the live drive, runs the REAL RunTier2 to mirror it onto the
// second drive, then returns a Manager ready to restore. The mirror's contents differ from the
// primary's afterwards when the caller mutates one of them — B1 and B2 both depend on being able to
// tell the two copies apart.
func r102Tier2Fixture(t *testing.T, volTars []string, dbDump string) *r102T2 {
t.Helper()
tmp := t.TempDir()
live := filepath.Join(tmp, "usb")
dest := filepath.Join(tmp, "flash")
sys := filepath.Join(tmp, "sys")
primaryUnit := r102UnitOnDrive(t, live, "primary", volTars, dbDump)
sett, err := settings.Load(filepath.Join(tmp, "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
if err := sett.AddStoragePath(settings.StoragePath{Path: dest, Label: "flash", Schedulable: true}); err != nil {
t.Fatal(err)
}
fake := &fakeRecoveryProvider{hdd: live, running: true}
cfg := &config.Config{}
cfg.Paths.SystemDataPath = sys
cfg.Paths.DataDir = filepath.Join(tmp, "data")
m := NewManager(cfg, sett, log.New(io.Discard, "", 0))
m.stackProvider = fake
m.systemDataPath = sys
m.tier2Mirror = copyTree
m.tier2SSDFits = func(string, int64) bool { return true }
m.samePhysicalDevice = oneDrivePerSubtree
if err := m.RunTier2("app"); err != nil {
t.Fatalf("RunTier2 (the code that writes the mirror this task reads): %v", err)
}
destBase := filepath.Join(dest, "backups", "secondary", "app")
if _, sErr := os.Stat(UnitManifestFile(tier2UnitDir(destBase))); sErr != nil {
t.Fatalf("Tier-2 did not mirror the unit — the fixture proves nothing: %v", sErr)
}
var vd, dp []string
m.volumeReplayFrom = func(_, dumpDir string) (int, error) {
vd = append(vd, dumpDir)
entries, rErr := os.ReadDir(dumpDir)
if rErr != nil {
return 0, nil
}
n := 0
for _, e := range entries {
if strings.HasSuffix(e.Name(), ".tar") {
n++
}
}
return n, nil
}
m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) {
return []DiscoveredDB{{StackName: "app", ContainerName: "app-db", DBType: DBTypePostgres}}, nil
}
m.importDBDump = func(_ context.Context, _ DiscoveredDB, p string) error {
dp = append(dp, p)
return nil
}
return &r102T2{m: m, fake: fake, liveDrive: live, destDrive: dest, destBase: destBase,
primaryUnit: primaryUnit, volDirs: &vd, dbPaths: &dp}
}
// markMirror rewrites the MIRROR's app.yaml so the two copies are distinguishable. It edits the copy,
// never the primary, so a restore that read the primary would return the pre-edit value.
func (f *r102T2) markMirror(t *testing.T, marker string) {
t.Helper()
p := filepath.Join(UnitComposeDir(tier2UnitDir(f.destBase)), "app.yaml")
b, err := os.ReadFile(p)
if err != nil {
t.Fatal(err)
}
out := strings.Replace(string(b), "SUBDOMAIN: primary", "SUBDOMAIN: "+marker, 1)
if out == string(b) {
t.Fatalf("mirror app.yaml did not carry the expected marker; contents:\n%s", b)
}
mustWrite(t, p, out)
}
// B1 — TestR102_Tier2UnitRestoreReadsTheSecondaryMirror.
func TestR102_Tier2UnitRestoreReadsTheSecondaryMirror(t *testing.T) {
f := r102Tier2Fixture(t, []string{"vol_a.tar"}, pgDump(1))
f.markMirror(t, "from-the-mirror")
res, err := f.m.RestoreTier2Unit("app")
if err != nil {
t.Fatalf("Tier-2 unit restore: %v", err)
}
if got := f.fake.gotEnv["SUBDOMAIN"]; got != "from-the-mirror" {
t.Errorf("config came from the PRIMARY unit: SUBDOMAIN=%q", got)
}
wantVol := UnitVolumeDumpDir(tier2UnitDir(f.destBase))
if len(*f.volDirs) != 1 || (*f.volDirs)[0] != wantVol {
t.Errorf("volume tars read from %v, want the mirror's %q", *f.volDirs, wantVol)
}
wantDB := UnitDBDumpDir(tier2UnitDir(f.destBase))
if len(*f.dbPaths) != 1 || filepath.Dir((*f.dbPaths)[0]) != wantDB {
t.Errorf("DB dump read from %v, want a file under the mirror's %q", *f.dbPaths, wantDB)
}
if res.VolumesReplayed != 1 || res.DBsReplayed != 1 {
t.Errorf("nothing came back: volumes=%d dbs=%d", res.VolumesReplayed, res.DBsReplayed)
}
if res.ManifestVolumes != 1 || res.ManifestDBs != 1 {
t.Errorf("the mirror's manifest did not enumerate its dumps: %+v", res)
}
}
// B2 — TestR102_Tier2UnitRestoreWorksWithThePrimaryUnitABSENT.
//
// THE ACCEPTANCE TEST. Tier-2 exists for the loss of the primary drive, and in that failure the
// primary recovery unit is GONE. A test that leaves the primary in place proves the code compiles,
// not that the mirror is a route (07-backup-architecture §7.2).
//
// The primary unit is not merely emptied — the whole `backups/` tree on the live drive is removed and
// then made unreadable, so any code that tried to fall back to it would error rather than silently
// find nothing.
//
// Red-proof (recorded in REPORT.md): point RestoreTier2Unit back at the primary unit path → this test
// fails with "no readable recovery unit ... falling back to volume-only restore" behaviour and the
// mirror's marker never reaching the redeploy.
func TestR102_Tier2UnitRestoreWorksWithThePrimaryUnitABSENT(t *testing.T) {
f := r102Tier2Fixture(t, []string{"vol_a.tar", "vol_b.tar"}, pgDump(1))
f.markMirror(t, "only-the-mirror-survives")
// The primary drive's whole backup tree is destroyed, then the parent is made unreadable so a
// fallback read cannot quietly return "nothing here".
primaryBackups := filepath.Join(f.liveDrive, "backups")
if err := os.RemoveAll(primaryBackups); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(primaryBackups, 0o000); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = os.Chmod(primaryBackups, 0o755) })
if _, err := os.Stat(f.primaryUnit); err == nil {
t.Fatalf("the primary unit is still readable at %q — the test would prove nothing", f.primaryUnit)
}
res, err := f.m.RestoreTier2Unit("app")
if err != nil {
t.Fatalf("the restore must succeed from the mirror ALONE — this is the whole point of R-102: %v", err)
}
if got := f.fake.gotEnv["SUBDOMAIN"]; got != "only-the-mirror-survives" {
t.Errorf("SUBDOMAIN=%q — the mirror was not the source", got)
}
if got := f.fake.gotEnv["SECRET_KEY"]; got != "key-primary" {
t.Errorf("the data-encrypting key did not come back from the mirrored unit: %q", got)
}
if res.VolumesReplayed != 2 {
t.Errorf("VolumesReplayed=%d, want 2 from the mirror", res.VolumesReplayed)
}
if res.DBsReplayed != 1 {
t.Errorf("DBsReplayed=%d, want 1 from the mirror", res.DBsReplayed)
}
wantVol := UnitVolumeDumpDir(tier2UnitDir(f.destBase))
if len(*f.volDirs) != 1 || (*f.volDirs)[0] != wantVol {
t.Errorf("volume source = %v, want %q", *f.volDirs, wantVol)
}
}
// B3 — TestR102_Tier2UnitRestoreTakesTheSingleWriterFlag. Every restore in this manager shares one
// running flag. The Tier-2 unit route must be inside it, not beside it (R-351b).
func TestR102_Tier2UnitRestoreTakesTheSingleWriterFlag(t *testing.T) {
f := r102Tier2Fixture(t, []string{"vol_a.tar"}, "")
// A backup/restore is already in flight.
if err := f.m.acquireRunning(); err != nil {
t.Fatal(err)
}
_, err := f.m.RestoreTier2Unit("app")
if err == nil {
t.Fatal("a second restore started while one was in flight")
}
if !strings.Contains(err.Error(), "already in progress") {
t.Errorf("refusal = %v, want the single-writer refusal", err)
}
if f.fake.stopped {
t.Error("the app was stopped by a restore that should never have started")
}
f.m.releaseRunning()
// And the flag is RELEASED afterwards, or the next restore would be refused forever.
if _, err := f.m.RestoreTier2Unit("app"); err != nil {
t.Fatalf("restore after the flag cleared: %v", err)
}
if err := f.m.acquireRunning(); err != nil {
t.Errorf("the running flag was not released after the Tier-2 unit restore: %v", err)
}
f.m.releaseRunning()
}
// B4 — TestR102_NoTier2CopyIsAnHonestRefusal. No recorded copy at all: refuse with the reason the
// file restore already uses, and take no outage.
func TestR102_NoTier2CopyIsAnHonestRefusal(t *testing.T) {
f := r102Tier2Fixture(t, []string{"vol_a.tar"}, "")
// Forget the recorded copy entirely — the "this app has never had a Tier-2 run" shape.
if err := f.m.settings.SetCrossDriveConfig("app", &settings.CrossDriveBackup{}); err != nil {
t.Fatal(err)
}
_, err := f.m.RestoreTier2Unit("app")
if err == nil {
t.Fatal("expected a refusal with no recorded copy")
}
if !strings.Contains(err.Error(), errNoTier2Copy.Error()) {
t.Errorf("refusal = %v, want errNoTier2Copy", err)
}
if f.fake.stopped {
t.Error("the app was stopped despite the refusal")
}
}
// B5 — TestR102_CopyAgeIsCarriedToTheSurface. The action overwrites live data with a copy of a
// certain age, so the age has to reach the surface. R-101 governs WHICH date: the success anchor,
// never the attempt clock, and Tier2CopyDate says which one it returned.
func TestR102_CopyAgeIsCarriedToTheSurface(t *testing.T) {
f := r102Tier2Fixture(t, []string{"vol_a.tar"}, "")
cov, err := f.m.Tier2RestoreCoverage("app")
if err != nil {
t.Fatalf("coverage: %v", err)
}
if cov.CopyLastSuccess == "" {
t.Fatal("the successful Tier-2 run left no LastSuccess for the surface to name")
}
date, proven := cov.Tier2CopyDate()
if !proven || date != cov.CopyLastSuccess {
t.Errorf("Tier2CopyDate() = (%q,%v), want the success anchor %q", date, proven, cov.CopyLastSuccess)
}
// R-101's other half: a later FAILED attempt must not become the date shown. LastRun advances,
// LastSuccess does not, and the surface must keep naming the copy that actually exists.
older := cov.CopyLastSuccess
if err := f.m.settings.UpdateCrossDriveStatus("app", func(c *settings.CrossDriveBackup) {
c.LastRun = "2099-01-01T00:00:00Z"
c.LastStatus = "error"
}); err != nil {
t.Fatal(err)
}
cov2, err := f.m.Tier2RestoreCoverage("app")
if err != nil {
t.Fatalf("coverage after a failed attempt: %v", err)
}
date2, proven2 := cov2.Tier2CopyDate()
if !proven2 || date2 != older {
t.Errorf("after a failed attempt the surface would name %q (proven=%v); want the last SUCCESS %q", date2, proven2, older)
}
}
// TestR102_Tier2UnitRestoreDoesNotWriteTheMirror — a restore reads its source. If the Tier-2 copy
// were mutated, the second drive would stop being a way back the moment it was used once.
func TestR102_Tier2UnitRestoreDoesNotWriteTheMirror(t *testing.T) {
f := r102Tier2Fixture(t, []string{"vol_a.tar"}, pgDump(1))
before := fingerprintTree(t, f.destBase)
if _, err := f.m.RestoreTier2Unit("app"); err != nil {
t.Fatalf("restore: %v", err)
}
if after := fingerprintTree(t, f.destBase); after != before {
t.Error("the Tier-2 copy was written to by a restore that only reads it")
}
}
@@ -0,0 +1,369 @@
package backup
import (
"context"
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
)
// R-102 Group A — the recovery-unit restore can be pointed at a unit ANYWHERE.
//
// The defect: every reader of a recovery unit could only name a path under `backups/primary/`
// (appbackup/paths.go joined it literally), while Tier-2 mirrored each app's whole unit to
// `<dest>/backups/secondary/<app>/recovery-unit/` on every run. So the mirror was written nightly for
// months and read by nothing — and it was unreadable in exactly the failure Tier-2 exists for, where
// the primary drive and its unit are gone (07-backup-architecture §6.3, §7.2).
//
// Every test below asserts WHICH DIRECTORY was read, not merely that a restore succeeded. A test that
// only checked `err == nil` would have passed against the pre-fix code, because the pre-fix code read
// a real unit — just never the one that survives.
// r102Unit captures a REAL recovery unit for "app" onto its own fresh drive, with an identifying
// SUBDOMAIN, a portable DB password and a portable data key, plus optional volume tars and a DB dump.
//
// The unit is produced by the production CaptureRecoveryUnit, not hand-written: the capture side and
// the restore side must meet at real bytes, so a change to the on-disk shape cannot pass by having a
// test agree with itself (the reason captureFixtureUnit is built the same way).
func r102Unit(t *testing.T, marker string, volTars []string, dbDump string) (drive, unitDir string) {
t.Helper()
tmp := t.TempDir()
drive = filepath.Join(tmp, "drive")
return drive, r102UnitOnDrive(t, drive, marker, volTars, dbDump)
}
// r102UnitOnDrive is r102Unit against a drive the caller already owns — the Tier-2 fixture needs the
// unit to sit on the SAME drive the Tier-2 run will mirror FROM.
func r102UnitOnDrive(t *testing.T, drive, marker string, volTars []string, dbDump string) (unitDir string) {
t.Helper()
tmp := t.TempDir()
stackDir := filepath.Join(tmp, "stack")
if err := os.MkdirAll(stackDir, 0o755); err != nil {
t.Fatal(err)
}
mustWrite(t, filepath.Join(stackDir, "docker-compose.yml"),
"services:\n app:\n image: example/app:1\n db:\n image: postgres:16\n")
mustWrite(t, filepath.Join(stackDir, ".felhom.yml"), "display_name: App\n")
mustWrite(t, filepath.Join(stackDir, "app.yaml"), "deployed: true\nenv:\n SUBDOMAIN: "+marker+"\n")
info := RecoveryInfo{
StackDir: stackDir,
DisplayName: "App",
ImagePins: []string{"example/app:1"},
NonSecretEnv: map[string]string{"SUBDOMAIN": marker},
SecretEnvVars: []string{"DB_PASSWORD", "SECRET_KEY"},
DataKeyEnvVars: []string{"SECRET_KEY"},
PortableSecretEnvVars: []string{"DB_PASSWORD", "SECRET_KEY"},
PortableSecrets: map[string]string{"DB_PASSWORD": "pw-" + marker, "SECRET_KEY": "key-" + marker},
}
unitDir = RecoveryUnitPath(drive, "app")
// The dumps are written BEFORE the capture so the manifest enumerates them — a manifest that
// lists nothing is the R-353 "the backup held only settings" shape, which is a different case.
for _, v := range volTars {
mustWrite(t, filepath.Join(UnitVolumeDumpDir(unitDir), v), "tar:"+v+":"+marker)
}
if dbDump != "" {
mustWrite(t, filepath.Join(UnitDBDumpDir(unitDir), "app-postgres.sql"), dbDump)
}
m := &Manager{
logger: log.New(io.Discard, "", 0),
systemDataPath: filepath.Join(tmp, "system"),
stackProvider: &fakeRecoveryProvider{info: info, hdd: drive},
version: "vtest",
}
if err := m.CaptureRecoveryUnit("app"); err != nil {
t.Fatalf("capture fixture %q: %v", marker, err)
}
return unitDir
}
// r102Manager builds a restore-side Manager whose LIVE drive is liveDrive, with the Docker and DB
// seams injected so no daemon is touched. The returned slices record the directories each leg read.
func r102Manager(t *testing.T, liveDrive string) (m *Manager, fake *fakeRecoveryProvider, volDirs, dbPaths *[]string) {
t.Helper()
fake = &fakeRecoveryProvider{hdd: liveDrive, running: true}
m = &Manager{
logger: log.New(io.Discard, "", 0),
systemDataPath: filepath.Join(liveDrive, "..", "sys"),
stackProvider: fake,
}
var vd, dp []string
m.volumeReplayFrom = func(_, dumpDir string) (int, error) {
vd = append(vd, dumpDir)
entries, err := os.ReadDir(dumpDir)
if err != nil {
return 0, nil
}
n := 0
for _, e := range entries {
if strings.HasSuffix(e.Name(), ".tar") {
n++
}
}
return n, nil
}
m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) {
return []DiscoveredDB{{StackName: "app", ContainerName: "app-db", DBType: DBTypePostgres}}, nil
}
m.importDBDump = func(_ context.Context, _ DiscoveredDB, p string) error {
dp = append(dp, p)
return nil
}
return m, fake, &vd, &dp
}
// A2 — TestR102_RestoreFromRecoveryUnitAtReadsTheGivenDir.
//
// Two complete units exist on disk with DIFFERENT contents. The restore is pointed at the second one
// and must read every leg — env, secrets, volume tars, DB dump — out of THAT directory. The first
// unit is the app's own primary unit and is present and perfectly readable, which is what makes the
// assertion mean something: a restore that ignored unitDir would still succeed, and would silently
// return the wrong copy's data.
func TestR102_RestoreFromRecoveryUnitAtReadsTheGivenDir(t *testing.T) {
primaryDrive, primaryUnit := r102Unit(t, "primary", []string{"vol_a.tar"}, pgDump(1))
_, mirrorUnit := r102Unit(t, "mirror", []string{"vol_a.tar", "vol_b.tar"}, pgDump(2))
m, fake, volDirs, dbPaths := r102Manager(t, primaryDrive)
res, err := m.RestoreFromRecoveryUnitAt("app", mirrorUnit)
if err != nil {
t.Fatalf("restore from the named unit: %v", err)
}
if fake.gotEnv == nil {
t.Fatal("recreate was never called — the restore did not reach the redeploy")
}
if got := fake.gotEnv["SUBDOMAIN"]; got != "mirror" {
t.Errorf("config came from the WRONG unit: SUBDOMAIN=%q, want %q", got, "mirror")
}
if got := fake.gotEnv["SECRET_KEY"]; got != "key-mirror" {
t.Errorf("data key came from the WRONG unit: %q", got)
}
if got := fake.gotEnv["DB_PASSWORD"]; got != "pw-mirror" {
t.Errorf("DB password came from the WRONG unit: %q", got)
}
if len(*volDirs) != 1 || (*volDirs)[0] != UnitVolumeDumpDir(mirrorUnit) {
t.Errorf("volume tars read from %v, want %q", *volDirs, UnitVolumeDumpDir(mirrorUnit))
}
if res.VolumesReplayed != 2 {
t.Errorf("VolumesReplayed=%d, want 2 (the mirror holds two tars; the primary holds one)", res.VolumesReplayed)
}
if len(*dbPaths) != 1 || filepath.Dir((*dbPaths)[0]) != UnitDBDumpDir(mirrorUnit) {
t.Errorf("DB dump read from %v, want a file under %q", *dbPaths, UnitDBDumpDir(mirrorUnit))
}
// The negative control: nothing was read out of the primary unit.
for _, d := range *volDirs {
if strings.HasPrefix(d, primaryUnit) {
t.Errorf("a leg was read from the PRIMARY unit %q despite a mirror being named", d)
}
}
}
// A3 — TestR102_PrimaryPathIsUnchanged. The one-argument wrapper must still resolve to the app's own
// drive, `backups/primary/<stack>`. Asserted on the DIRECTORIES the legs actually read, not on a
// path expression re-derived in the test.
func TestR102_PrimaryPathIsUnchanged(t *testing.T) {
primaryDrive, primaryUnit := r102Unit(t, "primary", []string{"vol_a.tar"}, pgDump(1))
m, fake, volDirs, dbPaths := r102Manager(t, primaryDrive)
if _, err := m.RestoreFromRecoveryUnit("app"); err != nil {
t.Fatalf("primary restore: %v", err)
}
if got := fake.gotEnv["SUBDOMAIN"]; got != "primary" {
t.Errorf("SUBDOMAIN=%q, want the primary unit's %q", got, "primary")
}
if !strings.Contains(primaryUnit, filepath.Join("backups", "primary", "app")) {
t.Fatalf("fixture is not where the primary unit belongs: %q", primaryUnit)
}
if len(*volDirs) != 1 || (*volDirs)[0] != UnitVolumeDumpDir(primaryUnit) {
t.Errorf("volume dir = %v, want %q", *volDirs, UnitVolumeDumpDir(primaryUnit))
}
if len(*dbPaths) != 1 || filepath.Dir((*dbPaths)[0]) != UnitDBDumpDir(primaryUnit) {
t.Errorf("db dump dir = %v, want %q", *dbPaths, UnitDBDumpDir(primaryUnit))
}
}
// A4 — TestR102_LiveDestinationIsUnchanged. THE SOURCE MOVES; THE DESTINATION DOES NOT.
//
// The restore reads a mirror that lives on a different drive entirely, and must still write the app
// back to its own live namespace: the definition through RecreateStackDefinitionFromUnit (whose
// compose dir must be the MIRROR's, since that is the definition being restored), and the data into
// the LIVE Docker volumes and the LIVE database container. A restore that also relocated the app's
// data would be a migration, not a restore.
//
// The observable for "the destination did not move" is that nothing under the mirror's own drive was
// written to: the mirror tree is byte-identical before and after.
func TestR102_LiveDestinationIsUnchanged(t *testing.T) {
primaryDrive, _ := r102Unit(t, "primary", []string{"vol_a.tar"}, pgDump(1))
mirrorDrive, mirrorUnit := r102Unit(t, "mirror", []string{"vol_a.tar"}, pgDump(2))
before := fingerprintTree(t, mirrorDrive)
m, fake, _, _ := r102Manager(t, primaryDrive)
var recreateComposeDir string
m.stackProvider = &r102RecordingProvider{fakeRecoveryProvider: fake, composeDirOut: &recreateComposeDir}
if _, err := m.RestoreFromRecoveryUnitAt("app", mirrorUnit); err != nil {
t.Fatalf("restore: %v", err)
}
if recreateComposeDir != UnitComposeDir(mirrorUnit) {
t.Errorf("recreate read compose from %q, want the mirror's %q", recreateComposeDir, UnitComposeDir(mirrorUnit))
}
if after := fingerprintTree(t, mirrorDrive); after != before {
t.Error("the SOURCE mirror was written to — a restore reads its source and never writes it")
}
// The live drive is where the app lives, and the restore must not have relocated it: the app's
// own drive path is still the one the manager resolves for it.
if got := m.GetAppDrivePath("app"); got != primaryDrive {
t.Errorf("the app's live drive moved to %q, want %q", got, primaryDrive)
}
}
// r102RecordingProvider captures the compose directory RecreateStackDefinitionFromUnit is handed —
// the one place the restore names a source directory to the guest side.
type r102RecordingProvider struct {
*fakeRecoveryProvider
composeDirOut *string
}
func (f *r102RecordingProvider) RecreateStackDefinitionFromUnit(name, composeDir string, fullEnv map[string]string) error {
*f.composeDirOut = composeDir
return f.fakeRecoveryProvider.RecreateStackDefinitionFromUnit(name, composeDir, fullEnv)
}
// A5 — TestR102_MutationOrderIsPreserved. R-47's ordering is pinned and R-102 must not have moved it:
// stop → volumes → recreate → DB-only start → replay → full start. The DB-only phase exists because
// starting the whole stack first let the application rebuild schema underneath the replay (H4,
// DIAG-immich-restore-round2-2026-07-19).
//
// The state AT REPLAY TIME is asserted, not just the call sequence — H4 looked correctly ordered and
// the app was up.
//
// Red-proof (recorded in REPORT.md): swap the volume replay and the recreate step in
// RestoreFromRecoveryUnitAt → this test fails on the sequence assertion.
func TestR102_MutationOrderIsPreserved(t *testing.T) {
primaryDrive, _ := r102Unit(t, "primary", nil, pgDump(1))
_, mirrorUnit := r102Unit(t, "mirror", []string{"vol_a.tar"}, pgDump(2))
m, fake, _, _ := r102Manager(t, primaryDrive)
var volumesDoneAtRecreate, fullUpAtReplay bool
var volumesReplayed bool
m.volumeReplayFrom = func(_, _ string) (int, error) {
volumesReplayed = true
if fake.gotEnv != nil {
t.Error("volumes were replayed AFTER the definition was recreated")
}
return 1, nil
}
prov := &r102OrderProvider{fakeRecoveryProvider: fake, volumesReplayed: &volumesReplayed, seen: &volumesDoneAtRecreate}
m.stackProvider = prov
m.importDBDump = func(context.Context, DiscoveredDB, string) error {
fullUpAtReplay = fake.fullStarted
return nil
}
if _, err := m.RestoreFromRecoveryUnitAt("app", mirrorUnit); err != nil {
t.Fatalf("restore: %v", err)
}
if got := strings.Join(fake.calls, ","); got != "stop,recreate,startsvc:db,start" {
t.Fatalf("sequence = %q, want stop → recreate → db-only start → (replay) → full start", got)
}
if !volumesDoneAtRecreate {
t.Error("the volume replay had not run when the definition was recreated")
}
if fullUpAtReplay {
t.Error("the FULL stack was already up when the DB replay fired — this is H4 exactly (R-47)")
}
}
// r102OrderProvider records whether the volume replay had already happened by the time the definition
// was recreated — the ordering fact the call log alone cannot carry.
type r102OrderProvider struct {
*fakeRecoveryProvider
volumesReplayed *bool
seen *bool
}
func (f *r102OrderProvider) RecreateStackDefinitionFromUnit(name, composeDir string, fullEnv map[string]string) error {
*f.seen = *f.volumesReplayed
return f.fakeRecoveryProvider.RecreateStackDefinitionFromUnit(name, composeDir, fullEnv)
}
// A6 — TestR102_MissingManifestInMirrorFailsClosed. A DIRECTORY IS NOT A PACKAGE.
//
// `<dest>/backups/secondary/<app>/recovery-unit/` can exist and be useless: a copy interrupted
// mid-run, or a tree whose manifest.json never landed. The Tier-2 unit restore must refuse it and
// must leave the app running — a restore armed over an unopenable unit would stop the app, replay
// nothing, and rewrite its definition from an empty capture. Same lesson as R-358 one tier over.
//
// Placed with Group A because it is the fail-closed half of the parameterised unit; the route it
// exercises is Part 1.3's.
func TestR102_MissingManifestInMirrorFailsClosed(t *testing.T) {
for _, tc := range []struct {
name string
setup func(t *testing.T, unitDir string)
}{
{"manifest absent", func(t *testing.T, unitDir string) {
if err := os.Remove(UnitManifestFile(unitDir)); err != nil {
t.Fatal(err)
}
}},
{"manifest unparseable", func(t *testing.T, unitDir string) {
mustWrite(t, UnitManifestFile(unitDir), "{ this is not json")
}},
{"recovery-unit absent entirely", func(t *testing.T, unitDir string) {
if err := os.RemoveAll(unitDir); err != nil {
t.Fatal(err)
}
}},
} {
t.Run(tc.name, func(t *testing.T) {
f := r102Tier2Fixture(t, []string{"vol_a.tar"}, pgDump(1))
tc.setup(t, tier2UnitDir(f.destBase))
cov, covErr := f.m.Tier2RestoreCoverage("app")
if covErr != nil {
t.Fatalf("coverage: %v", covErr)
}
if cov.CanRestoreUnit() {
t.Error("CanRestoreUnit() said yes over an unopenable mirror — the surface would offer the action")
}
_, err := f.m.RestoreTier2Unit("app")
if err == nil {
t.Fatal("the restore ran over an unopenable mirror")
}
if !strings.Contains(err.Error(), ErrTier2NoUnitInCopy.Error()) {
t.Errorf("refusal = %v, want ErrTier2NoUnitInCopy", err)
}
if f.fake.stopped {
t.Error("the app was STOPPED despite the refusal — the outage this gate exists to avoid")
}
if f.fake.gotEnv != nil {
t.Error("the app's definition was rewritten despite the refusal")
}
})
}
}
// TestR102_UnopenableMirrorIsStillDisclosedAsUnread is the other side of A6, and the reason
// UnitRestorable is a second field rather than a widening of HasUnit: a half-copied mirror cannot be
// restored, but it IS captured data the file restore is not looking at, so the disclosure must stand.
func TestR102_UnopenableMirrorIsStillDisclosedAsUnread(t *testing.T) {
f := r102Tier2Fixture(t, []string{"vol_a.tar"}, pgDump(1))
mustWrite(t, UnitManifestFile(tier2UnitDir(f.destBase)), "{ not json")
cov, err := f.m.Tier2RestoreCoverage("app")
if err != nil {
t.Fatalf("coverage: %v", err)
}
if !cov.HasUnit {
t.Error("HasUnit went false for a mirror that exists — the file restore would stop disclosing unread data")
}
if cov.UnitRestorable {
t.Error("UnitRestorable stayed true over an unparseable manifest")
}
}
@@ -0,0 +1,103 @@
package backup
import (
"os"
"path/filepath"
"testing"
)
// R-103 Group C — the ADDITIVE file restore is untouched.
//
// R-102 gave the Tier-2 copy a second, DESTRUCTIVE action. The file restore beside it is proven live
// (Campaign 9 A1 and A3, 39 s and 46 s, byte-identical returns) and its promises are load-bearing:
// nothing live is overwritten and nothing is deleted. This group is the non-regression assertion, and
// it is written first-class rather than as an afterthought, because a silent widening here would turn
// „restore my deleted files" into „overwrite everything with last night's copy".
// C1 — TestR103_CanRestoreStillAnswersLegsOnly.
//
// CanRestore() gates the additive restore and must keep answering exactly one question. Widening it
// to include the unit is R-356 arriving a second time: there, ONE predicate meant both "has this app
// a drive?" and "is this app installed?", and it refused 40 running apps for months.
//
// Red-proof (recorded in REPORT.md): make CanRestore return `len(c.Legs) > 0 || c.HasUnit` → the
// unit-only case below fails.
func TestR103_CanRestoreStillAnswersLegsOnly(t *testing.T) {
cases := []struct {
name string
cov Tier2Coverage
want bool
}{
{"no legs, no unit", Tier2Coverage{}, false},
{"no legs, unit present", Tier2Coverage{HasUnit: true}, false},
{"no legs, unit RESTORABLE", Tier2Coverage{HasUnit: true, UnitRestorable: true}, false},
{"legs present", Tier2Coverage{Legs: []string{"hdd"}}, true},
{"legs and unit", Tier2Coverage{Legs: []string{"hdd", "userdata"}, HasUnit: true, UnitRestorable: true}, true},
}
for _, c := range cases {
if got := c.cov.CanRestore(); got != c.want {
t.Errorf("%s: CanRestore() = %v, want %v", c.name, got, c.want)
}
}
// And the second predicate answers its own question, independently of the first.
if (Tier2Coverage{Legs: []string{"hdd"}}).CanRestoreUnit() {
t.Error("CanRestoreUnit() went true on file legs alone — the two predicates are entangled")
}
if !(Tier2Coverage{UnitRestorable: true}).CanRestoreUnit() {
t.Error("CanRestoreUnit() went false on a restorable unit")
}
}
// C2 — TestR103_FileRestoreBehaviourUnchanged. The additive restore's COUNTS and its two
// non-destruction promises, over a copy that also holds a restorable unit — the case R-102 created
// and the one where a leak between the paths would show.
func TestR103_FileRestoreBehaviourUnchanged(t *testing.T) {
m, fake, liveDrive, destDrive := newT2RManager(t)
destBase := filepath.Join(destDrive, "backups", "secondary", "app")
// The copy holds BOTH: a file leg and a full, openable recovery unit.
mustWrite(t, filepath.Join(destBase, "hdd", "appdata", "photos", "gone.jpg"), "RESTORED")
mustWrite(t, filepath.Join(destBase, "hdd", "appdata", "photos", "kept.jpg"), "FROM-THE-BACKUP")
unit := tier2UnitDir(destBase)
mustWrite(t, UnitManifestFile(unit), `{"schema_version":1,"app_name":"app"}`)
mustWrite(t, filepath.Join(UnitVolumeDumpDir(unit), "vol_a.tar"), "tar")
// Live: one of the two files exists with DIFFERENT content, and a third file exists only live.
liveDir := filepath.Join(liveDrive, "appdata", "photos")
mustWrite(t, filepath.Join(liveDir, "kept.jpg"), "THE-CUSTOMERS-NEWER-EDIT")
mustWrite(t, filepath.Join(liveDir, "only-live.jpg"), "NOT-IN-THE-BACKUP")
cov, err := m.Tier2RestoreCoverage("app")
if err != nil {
t.Fatalf("coverage: %v", err)
}
if !cov.CanRestoreUnit() {
t.Fatal("fixture is wrong: the unit must be restorable, or this proves nothing")
}
n, err := m.RestoreTier2Files("app")
if err != nil {
t.Fatalf("file restore: %v", err)
}
// Two: `gone.jpg` above and `a.jpg` from the shared fixture. NOT three — `kept.jpg` exists live
// and is skipped — and NOT more, which is what a leak from the unit's volume tars would look like.
if n != 2 {
t.Errorf("filesRestored = %d, want 2 (the two MISSING files only) — the file restore's reach changed", n)
}
if got, _ := os.ReadFile(filepath.Join(liveDir, "gone.jpg")); string(got) != "RESTORED" {
t.Errorf("the missing file did not come back: %q", got)
}
if got, _ := os.ReadFile(filepath.Join(liveDir, "kept.jpg")); string(got) != "THE-CUSTOMERS-NEWER-EDIT" {
t.Errorf("an EXISTING live file was overwritten: %q — the additive promise is broken", got)
}
if _, sErr := os.Stat(filepath.Join(liveDir, "only-live.jpg")); sErr != nil {
t.Error("a live file absent from the backup was DELETED — the second non-destruction promise is broken")
}
if len(fake.stopped) != 1 || len(fake.started) != 1 {
t.Errorf("lifecycle changed: stopped=%v started=%v", fake.stopped, fake.started)
}
// The unit the file restore does not read is untouched by it.
if got, _ := os.ReadFile(filepath.Join(UnitVolumeDumpDir(unit), "vol_a.tar")); string(got) != "tar" {
t.Error("the file restore wrote into the copy's recovery unit")
}
}
+16 -1
View File
@@ -89,10 +89,25 @@ func (m *Manager) reimportDBDumpsFrom(ctx context.Context, stackName, dumpDir st
return imported, nil
}
// dbReimportTimeout bounds a DB replay so a stuck import cannot hang a restore indefinitely. Named
// once because both bounded entry points below must agree: two paths that differ in how long they
// let a wedged import hold the restore are two different products (R-102 added the second one).
const dbReimportTimeout = 35 * time.Minute
// reimportDBDumpsCtx is a small helper that runs reimportDBDumps with a bounded context so a stuck DB
// import cannot hang the restore indefinitely.
func (m *Manager) reimportDBDumpsCtx(stackName, nsRoot string) (int, error) {
ctx, cancel := context.WithTimeout(context.Background(), 35*time.Minute)
ctx, cancel := context.WithTimeout(context.Background(), dbReimportTimeout)
defer cancel()
return m.reimportDBDumps(ctx, stackName, nsRoot)
}
// reimportDBDumpsAtCtx is reimportDBDumpsCtx with an EXPLICIT dump directory — the bounded-context
// twin of reimportDBDumpsFrom, added for R-102 so the Tier-2 unit restore can replay out of the
// secondary mirror. Same discovery/import seams, same failure semantics, same bound; only the source
// directory differs.
func (m *Manager) reimportDBDumpsAtCtx(stackName, dumpDir string) (int, error) {
ctx, cancel := context.WithTimeout(context.Background(), dbReimportTimeout)
defer cancel()
return m.reimportDBDumpsFrom(ctx, stackName, dumpDir)
}
+55 -9
View File
@@ -178,7 +178,36 @@ type UnitRestoreResult struct {
// D5: this no longer needs the guest. A restore with the guest's app.yaml absent succeeds, which is
// pinned by TestRestoreFromRecoveryUnitWithGuestAbsent — the withheld class is regenerated (O4) and
// only a data key missing from BOTH sources still refuses.
//
// R-102: this is now the thin caller. It names the PRIMARY unit — the app's own drive,
// backups/primary/<stack> — and hands it to RestoreFromRecoveryUnitAt, which holds the whole body.
// An unresolvable drive path is still refused inside …At, in the same place and with the same
// message, so the order of the checks a caller can observe is unchanged.
func (m *Manager) RestoreFromRecoveryUnit(stackName string) (UnitRestoreResult, error) {
return m.RestoreFromRecoveryUnitAt(stackName, RecoveryUnitPath(m.namespaceRoot(m.GetAppDrivePath(stackName)), stackName))
}
// RestoreFromRecoveryUnitAt is RestoreFromRecoveryUnit with an EXPLICIT recovery-unit directory.
//
// R-102. ONE implementation, two callers — the same rule restoreDockerVolumesFrom states beside
// itself in restore.go, and for the same reason: a second copy of this body is exactly how the local
// path and the off-site path drifted apart until nothing compared them.
//
// The reason it exists: Tier-2 mirrors the app's whole recovery unit to
// <dest>/backups/secondary/<stack>/recovery-unit/ on every run, and until now every reader of a unit
// could only name a path under backups/primary/. So in the one failure Tier-2 exists for — the
// primary drive is lost, taking the primary unit with it — the surviving copy could not be opened by
// any action in the product (07-backup-architecture §6.3, §7.2).
//
// THE SOURCE MOVES; THE DESTINATION DOES NOT. unitDir changes only where the manifest, the compose
// capture, the .sql dumps and the volume tars are READ from. The app's data is written back to the
// live Docker volumes and the live database container, and its definition to the guest, exactly as
// before — a restore that also relocated the app's data would be a migration, not a restore.
//
// Everything else is pinned and unchanged: the mutation order (stop → volumes → recreate → DB-only
// start → replay → start, R-47), the secret reconciliation with unit-over-guest precedence and the
// fail-closed data-key gate, and the no-unit fallback to RestoreApp with its CountsUnknown handling.
func (m *Manager) RestoreFromRecoveryUnitAt(stackName, unitDir string) (UnitRestoreResult, error) {
var res UnitRestoreResult
if m.stackProvider == nil {
return res, fmt.Errorf("stack provider not configured")
@@ -197,15 +226,18 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) (UnitRestoreResult,
m.mu.Unlock()
}()
// The DESTINATION side, and it is deliberately still resolved here: RestoreApp (the no-unit
// fallback below) needs it, and 07-backup-architecture §6.3 records that the restore destination
// is resolved by the same rule as the capture destination. It is no longer used to derive any
// SOURCE path — that is what unitDir is for.
drivePath := m.GetAppDrivePath(stackName)
if drivePath == "" || !filepath.IsAbs(drivePath) {
return res, fmt.Errorf("cannot determine drive path for %s", stackName)
}
nsRoot := m.namespaceRoot(drivePath)
manifest := readManifest(RecoveryUnitManifestPath(nsRoot, stackName))
manifest := readManifest(UnitManifestFile(unitDir))
if manifest == nil {
m.logger.Printf("[WARN] [backup] No recovery unit for %s — falling back to volume-only restore", stackName)
m.logger.Printf("[WARN] [backup] No readable recovery unit for %s at %s — falling back to volume-only restore", stackName, unitDir)
m.mu.Lock()
m.running = false // RestoreApp re-acquires the running flag
m.mu.Unlock()
@@ -220,7 +252,7 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) (UnitRestoreResult,
// was just parsed above, so the claim and the outcome are counted from the same document.
res.ManifestVolumes, res.ManifestDBs = len(manifest.VolumeDumps), len(manifest.DBDumps)
composeDir := RecoveryUnitComposePath(nsRoot, stackName)
composeDir := UnitComposeDir(unitDir)
nonSecretEnv, unitSecrets := readUnitEnv(filepath.Join(composeDir, "app.yaml"), manifest.PortableSecretEnvVars)
// D5: the unit carries the portable class, so this is the leg that no longer needs the guest. The
@@ -275,8 +307,11 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) (UnitRestoreResult,
stackName, len(unresolved), unresolved)
}
}
m.logger.Printf("[INFO] [backup] Restoring %s from recovery unit: images=%d, secrets recovered=%d/%d, data_keys=%d",
stackName, len(manifest.ImagePins), len(manifest.SecretEnvVars)-len(missing), len(manifest.SecretEnvVars), len(manifest.DataKeyEnvVars))
// R-102: the unit DIRECTORY is logged. Which copy a restore read from is now a real question with
// two answers, and "an absent log line is not evidence" — the drill reads this line to prove the
// secondary mirror, not the primary unit, was the source. It is a path, never a secret.
m.logger.Printf("[INFO] [backup] Restoring %s from recovery unit %s: images=%d, secrets recovered=%d/%d, data_keys=%d",
stackName, unitDir, len(manifest.ImagePins), len(manifest.SecretEnvVars)-len(missing), len(manifest.SecretEnvVars), len(manifest.DataKeyEnvVars))
// R-47: which compose service holds the database, and is there anything to replay? Resolved from
// the UNIT's compose, because that file is about to BECOME the live one. Both answers are needed
@@ -286,7 +321,8 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) (UnitRestoreResult,
// "cannot tell" is not "no database" — leave it empty and let the gate decide.
m.logger.Printf("[WARN] [backup] %s: could not read the unit's compose services: %v", stackName, dsErr)
}
hasDumps := hasReplayableDump(AppDBDumpPath(nsRoot, stackName))
dbDumpDir := UnitDBDumpDir(unitDir)
hasDumps := hasReplayableDump(dbDumpDir)
if hasDumps && len(dbServices) == 0 {
m.logger.Printf("[ERROR] [backup] Restore REFUSED for %s: a .sql dump exists but no database service is identifiable in the unit's compose", stackName)
return res, fmt.Errorf("Az adatbázis-szolgáltatás nem azonosítható a(z) %s alkalmazásban — a visszaállítás biztonsági okból nem indult el.", stackName)
@@ -302,7 +338,17 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) (UnitRestoreResult,
// R-353: the count is captured even when the replay errors — a partial replay is a fact the
// customer's sentence has to be built from, and discarding it on the error path is how Scenario C
// would end up wearing Scenario B's wording.
replayed, volErr := m.restoreDockerVolumes(stackName, drivePath)
// R-354's volume-REPLAY seam, reused here rather than a second one being invented. It defaults to
// the real restoreDockerVolumesFrom, so production behaviour is byte-for-byte what it was; what it
// buys is that R-102's acceptance test can assert WHICH directory the tars came out of without a
// Docker daemon. For 40 of the 53 catalogue apps that archive is the entire dataset, so "the
// mirror was the source" has to be provable for the volume leg too, not only for the env and the
// database.
volReplay := m.volumeReplayFrom
if volReplay == nil {
volReplay = m.restoreDockerVolumesFrom
}
replayed, volErr := volReplay(stackName, UnitVolumeDumpDir(unitDir))
res.VolumesReplayed = replayed
if volErr != nil {
m.logger.Printf("[ERROR] [backup] volume restore for %s: %v", stackName, volErr)
@@ -322,7 +368,7 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) (UnitRestoreResult,
if dataErr == nil {
dataErr = err
}
} else if n, err := m.reimportDBDumpsCtx(stackName, nsRoot); err != nil {
} else if n, err := m.reimportDBDumpsAtCtx(stackName, dbDumpDir); err != nil {
res.DBsReplayed = n // partial credit: whatever imported before the failure really did import
m.logger.Printf("[ERROR] [backup] DB re-import for %s: %v", stackName, err)
if dataErr == nil {
+118 -6
View File
@@ -39,6 +39,12 @@ var (
// are in this class. Exported so the handler can refuse BEFORE stopping the app and name the action
// that does work, instead of taking an outage and reporting "no missing files".
ErrTier2NoRestorableData = errors.New("ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza")
// ErrTier2NoUnitInCopy (R-102) — the recorded Tier-2 copy holds no OPENABLE recovery unit: either
// recovery-unit/ is absent, or it is a directory without a readable manifest.json. Exported so the
// handler can refuse before beginning any op. FAIL CLOSED is the whole point of the second half:
// a directory that exists is not a package, and reading a half-copied mirror as if it were one is
// how a restore would overwrite live data with nothing.
ErrTier2NoUnitInCopy = errors.New("a másodlagos másolatban nincs megnyitható mentési egység ehhez az alkalmazáshoz")
)
// Tier2Coverage says what a Tier-2 restore can and cannot return for one app — the asymmetry C9-F1
@@ -50,14 +56,64 @@ var (
// tarballs — which this restore path never opens. An app can have HasUnit && no Legs (43 of 53), in
// which case the restore is a guaranteed no-op no matter how much data was lost.
type Tier2Coverage struct {
Legs []string // subtrees this restore reads and that exist in the copy: "hdd", "userdata"
HasUnit bool // recovery-unit/ present — captured, but NOT restorable by this path
Legs []string // subtrees the FILE restore reads and that exist in the copy: "hdd", "userdata"
HasUnit bool // recovery-unit/ present as a DIRECTORY — the disclosure fact, see below
// UnitRestorable (R-102) — the mirror is a real PACKAGE, not merely a directory: recovery-unit/
// exists AND carries a manifest.json that parses. This is the gate for the UNIT restore.
//
// It is a SECOND FIELD and not a widening of HasUnit, and the distinction is load-bearing in both
// directions. HasUnit answers "is there captured data this FILE restore is not looking at?" — the
// question tier2UnitNotCoveredMsg is appended for, and the honest answer for a half-copied mirror
// is still yes. UnitRestorable answers "can the unit restore open this?" — and for that same
// half-copied mirror the answer is no. Collapsing them would either silence a true disclosure or
// arm a restore over an unopenable package.
UnitRestorable bool
// CopyLastRun / CopyLastSuccess — WHEN the copy this restore would read was written, so the
// surface can name the date before it overwrites anything with it (Scenario E). Filled only by
// Tier2RestoreCoverage, which is the path that holds the settings; tier2CoverageAt is a pure
// filesystem inspection and leaves them empty. They are STRINGS in the recorded RFC3339 form,
// carried verbatim — no formatting decision is taken in this package.
//
// R-101 applies here exactly as it does on the backup card: CopyLastRun is the ATTEMPT clock and
// CopyLastSuccess is the only evidence a copy was actually made. A surface that shows one must
// not present it as the other.
CopyLastRun string
CopyLastSuccess string
}
// CanRestore reports whether the restore has any subtree to read at all.
// CanRestore reports whether the FILE restore has any subtree to read at all.
//
// R-102/R-103: this answers exactly one question and must keep answering only that one. Widening it
// to include the unit is R-356 arriving a second time — there, ONE predicate meant both "has this app
// a drive?" and "is this app installed?", and it refused 40 running apps for months while telling
// their owners to reinstall them somewhere those apps never offer. Two questions, two predicates.
func (c Tier2Coverage) CanRestore() bool { return len(c.Legs) > 0 }
// tier2CoverageAt inspects a resolved copy directory. Pure filesystem stat — no side effects.
// CanRestoreUnit reports whether the UNIT restore can run from this copy — the second predicate.
func (c Tier2Coverage) CanRestoreUnit() bool { return c.UnitRestorable }
// tier2UnitDir returns the recovery-unit directory inside a resolved Tier-2 copy. ONE expression of
// where Tier-2 puts the mirror; tier2.go writes it at the same relative name ("Unit leg (always)").
func tier2UnitDir(destBase string) string {
return filepath.Join(destBase, "recovery-unit")
}
// tier2UnitIsOpenable reports whether a mirrored unit directory is a PACKAGE and not just a
// directory. Fail-closed by construction: the manifest must be present AND parse (readManifest
// returns nil for both a missing file and malformed JSON), because an unopenable unit that armed a
// restore would stop the app, replay nothing, and rewrite its definition from an empty capture.
//
// This is the R-358 lesson one tier over — the off-site scratch marker — arriving on Tier-2.
func tier2UnitIsOpenable(unitDir string) bool {
if fi, err := os.Stat(unitDir); err != nil || !fi.IsDir() {
return false
}
return readManifest(UnitManifestFile(unitDir)) != nil
}
// tier2CoverageAt inspects a resolved copy directory. Pure filesystem stat/read — no side effects.
func tier2CoverageAt(destBase string) Tier2Coverage {
var c Tier2Coverage
for _, leg := range []string{"hdd", "userdata"} {
@@ -65,9 +121,11 @@ func tier2CoverageAt(destBase string) Tier2Coverage {
c.Legs = append(c.Legs, leg)
}
}
if fi, err := os.Stat(filepath.Join(destBase, "recovery-unit")); err == nil && fi.IsDir() {
unitDir := tier2UnitDir(destBase)
if fi, err := os.Stat(unitDir); err == nil && fi.IsDir() {
c.HasUnit = true
}
c.UnitRestorable = tier2UnitIsOpenable(unitDir)
return c
}
@@ -79,7 +137,61 @@ func (m *Manager) Tier2RestoreCoverage(stackName string) (Tier2Coverage, error)
if err != nil {
return Tier2Coverage{}, err
}
return tier2CoverageAt(destBase), nil
cov := tier2CoverageAt(destBase)
// R-102: carry WHEN the copy was written, so the surface can name the date on an action that
// overwrites live data with it. Read from the same recorded config tier2RecordedCopyDir just
// resolved the path from, so the date and the directory cannot describe different runs.
if m.settings != nil {
if cfg := m.settings.GetCrossDriveConfig(stackName); cfg != nil {
cov.CopyLastRun, cov.CopyLastSuccess = cfg.LastRun, cfg.LastSuccess
}
}
return cov, nil
}
// RestoreTier2Unit runs the FULL recovery-unit restore from the app's Tier-2 copy on the SECOND
// DRIVE — R-102, and the reason this task exists.
//
// Tier-2 has mirrored each app's whole recovery unit to
// <dest>/backups/secondary/<app>/recovery-unit/ on every run for months, and no code path read it:
// every reader of a unit could only name a path under backups/primary/. So in the exact failure
// Tier-2 exists for — the primary drive is lost, and the primary unit with it — the surviving copy
// was unopenable by any customer action (07-backup-architecture §6.3, §7.2).
//
// It is NOT the additive file restore beside it. This one OVERWRITES: named volumes are recreated
// from the mirror's tars and the database is replayed from the mirror's dump. The surface must carry
// that difference; see tier2UnitConfirm in the web package.
//
// THE SINGLE-WRITER FLAG IS TAKEN INSIDE RestoreFromRecoveryUnitAt, exactly as on the primary path —
// do NOT add an acquireRunning() here. A second acquire would refuse the restore it is guarding.
func (m *Manager) RestoreTier2Unit(stackName string) (UnitRestoreResult, error) {
destBase, err := m.tier2RecordedCopyDir(stackName)
if err != nil {
return UnitRestoreResult{}, err
}
unitDir := tier2UnitDir(destBase)
if !tier2UnitIsOpenable(unitDir) {
m.logger.Printf("[WARN] [backup] Tier-2 unit restore refused for %s: no openable recovery unit in the recorded copy — the app was NOT stopped", stackName)
return UnitRestoreResult{}, ErrTier2NoUnitInCopy
}
// WARN, not INFO: this is the destructive one of the two Tier-2 restores. The unit directory is a
// path and never a secret, and naming it is what makes "the SECONDARY mirror was the source" a
// positive observable in the log rather than an absence to be argued from.
m.logger.Printf("[WARN] [backup] Tier-2 UNIT restore for %s from the secondary mirror %s — this OVERWRITES live app data", stackName, unitDir)
return m.RestoreFromRecoveryUnitAt(stackName, unitDir)
}
// Tier2CopyDate returns the date the surface should name for this app's Tier-2 copy, preferring the
// last SUCCESS over the last ATTEMPT (R-101: a timestamp that records "we tried" cannot answer "did
// it work"), and reports whether the returned value is a proven success.
//
// It exists so the confirm text and the outcome sentence cannot disagree about which copy is being
// restored: one resolver, two readers.
func (c Tier2Coverage) Tier2CopyDate() (date string, proven bool) {
if c.CopyLastSuccess != "" {
return c.CopyLastSuccess, true
}
return c.CopyLastRun, false
}
// tier2RecordedCopyDir resolves the RECORDED Tier-2 copy dir for a stack, applying every
+17 -7
View File
@@ -309,13 +309,12 @@ func (s *Server) templateFuncMap() template.FuncMap {
// moment the customer decides whether to restore — a UTC machine timestamp is not something a
// customer can reason about. Absolute rather than relative here on purpose: "3 napja" is fine on
// a status card, but a restore decision deserves the actual date.
"fmtTimeStr": func(s string) string {
t, err := time.Parse(time.RFC3339, s)
if err != nil {
return s
}
return t.In(loc).Format("2006-01-02 15:04")
},
//
// R-102: it delegates to the package-level fmtRFC3339Local so a HANDLER can render the same
// date the same way. The Tier-2 unit restore names the copy's date in its confirm (template)
// and again in its outcome (Go); two renderings of one decision that could disagree is how a
// customer ends up confirming one date and being told another.
"fmtTimeStr": fmtRFC3339Local,
"fmtTime": func(t time.Time) string {
if t.IsZero() {
return "–"
@@ -497,3 +496,14 @@ func (s *Server) templateFuncMap() template.FuncMap {
},
}
}
// fmtRFC3339Local renders an RFC3339 STRING as an absolute Budapest-local date-time, returning the
// input unchanged when it does not parse. The ONE implementation behind the `fmtTimeStr` template
// helper and every Go-side caller. See the comment on the funcmap entry for why it is shared.
func fmtRFC3339Local(s string) string {
t, err := time.Parse(time.RFC3339, s)
if err != nil {
return s
}
return t.In(getTimezone()).Format("2006-01-02 15:04")
}
+207 -9
View File
@@ -1187,6 +1187,23 @@ type AppBackupRow struct {
Tier2StatusBadge string // "Sikeres", "Hiba", "Fut...", "—"
Tier2SizeHuman string
// R-102/R-103 — the SECOND predicate, and it is a second field on purpose.
//
// Tier2UnitRestorable the copy holds an OPENABLE recovery unit (manifest present and parseable)
// → the destructive „Teljes visszaállítás" action is offered
// Tier2CopyDate the RFC3339 stamp of the copy that action would write over live data with
// Tier2CopyDateProven false = that stamp is only an ATTEMPT clock, never a proven copy (R-101)
//
// Tier2UnitRestorable is NOT derived from Tier2LastStatus, Tier2SizeHuman or anything else on this
// row: it is computed from what the copy on disk actually holds, because a row that says a backup
// succeeded is not evidence that the package inside it can be opened.
Tier2UnitRestorable bool
Tier2CopyDate string
Tier2CopyDateProven bool
// Tier2UnitConfirm is the assembled destructive-confirm sentence (tier2UnitConfirmMsg). Built in
// Go, not in the attribute, so it is one named string a test can assert verbatim.
Tier2UnitConfirm string
// Drive disconnected — app's home drive is currently disconnected
DriveDisconnected bool
// Tier2 destination drive is currently disconnected (backup paused, not failed)
@@ -1404,6 +1421,16 @@ func (s *Server) buildAppBackupRows(status *backup.FullBackupStatus) []AppBackup
row.Tier2LastError = cd.LastError
row.Tier2LastWarning = cd.LastWarning
row.Tier2SizeHuman = cd.LastSizeHuman
// R-102: ask the COPY, not the config. Tier2RestoreCoverage stats the recorded copy
// and reads its manifest, and it raises the same refusals the restore itself would —
// a disconnected destination, a pre-v2 layout — so an offer is never rendered for a
// copy the action would refuse. On any refusal the action is simply not offered; the
// row keeps rendering everything else it already showed.
if cov, covErr := s.backupMgr.Tier2RestoreCoverage(app.StackName); covErr == nil {
row.Tier2UnitRestorable = cov.CanRestoreUnit()
row.Tier2CopyDate, row.Tier2CopyDateProven = cov.Tier2CopyDate()
row.Tier2UnitConfirm = tier2UnitConfirmMsg(row.Tier2CopyDate, row.Tier2CopyDateProven)
}
switch cd.LastStatus {
case "ok":
row.Tier2StatusBadge = "Sikeres"
@@ -1580,18 +1607,102 @@ const monitoringIntegritySchedule = "Hetente, kimarado ellenorzest potol"
// verbatim by tests — a silent edit to either is the way an honest message drifts back into a
// comforting one.
const (
// tier2NoCoverageMsg is shown when this app's data cannot come from the secondary copy at all.
// It NAMES the action that works rather than leaving a dead end: the keep-side recovery-unit
// restore on /backups/restore, which does restore named volumes and DB dumps (proven live,
// Campaign 9 A2). It also states plainly that no outage was taken, because the previous behaviour
// took one.
// tier2NoCoverageMsg is shown when this app's data cannot come from the secondary copy AT ALL —
// no file legs AND no openable recovery unit in the copy. It NAMES the action that works rather
// than leaving a dead end: the keep-side recovery-unit restore on /backups/restore, which does
// restore named volumes and DB dumps (proven live, Campaign 9 A2). It also states plainly that no
// outage was taken, because the previous behaviour took one.
//
// R-103 NARROWED IT. Until v0.229.0 this same sentence was also shown to the far commoner case —
// no file legs but a full unit mirror sitting in the copy — and it sent those customers to a
// button on another page for data that is now restorable on the page they are already looking at.
// tier2UnitAvailableMsg is that case now.
tier2NoCoverageMsg = "Ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza — az alkalmazás nem állt le. Használd a Visszaállítás indítása gombot a Biztonsági mentés → Visszaállítás oldalon."
// tier2UnitNotCoveredMsg is appended wherever the restore DID run, so a clean result never reads
// as a clean bill of health for data the operation never opened.
// tier2UnitAvailableMsg (R-103) — the copy holds no restorable FILES, but it does hold an openable
// recovery unit, so the answer is the action beside this one, not a different page. It names the
// button by its own label and says why the two differ, because the difference is the whole reason
// they are not one button: this one overwrites.
tier2UnitAvailableMsg = "Ennek az alkalmazásnak az adatai nem fájlokban, hanem az alkalmazás saját adatbázisában és köteteiben vannak — az alkalmazás nem állt le. Ezeket a mellette lévő „Teljes visszaállítás a másolatból” gombbal tudod visszahozni ugyanerről a másolatról. Figyelem: az a művelet FELÜLÍRJA a jelenlegi adatokat, míg ez a gomb csak a hiányzó fájlokat pótolja."
// tier2UnitNotCoveredMsg is appended wherever the FILE restore DID run, so a clean result never
// reads as a clean bill of health for data the operation never opened.
//
// R-103: it is NOT deleted now that the unit is restorable. It is appended where the FILE restore
// ran, and it is still exactly true of that restore — the file merge still never opens the
// database or the named volumes. Deleting it would let a clean file-restore result read as a clean
// bill of health for data the operation did not look at, which is the sentence it exists to
// prevent.
tier2UnitNotCoveredMsg = "Az alkalmazás adatbázisa és belső kötetei nem tartoznak ebbe a visszaállításba."
// The Tier-2 UNIT restore's outcome suffix (R-102, Scenario E). The outcome names WHICH copy was
// just written over the app's live data — an action that overwrites must say what it overwrote
// with, in the sentence the customer is left holding.
tier2UnitRestoreSourceMsgFmt = "A visszaállítás forrása a második meghajtón lévő másolat volt (%s)."
// …and the R-101 variant. CopyLastRun is the ATTEMPT clock: it advances on a FAILED Tier-2 run
// too. Where no success has ever been recorded for this app, the date shown is evidence that a
// copy was attempted and nothing more, and the sentence must not present it as evidence of a copy.
tier2UnitRestoreSourceUnprovenMsgFmt = "A visszaállítás forrása a második meghajtón lévő másolat volt (%s — ez az utolsó mentési kísérlet ideje; azt nem tudjuk igazolni, hogy az sikeres volt)."
// R-102/R-103 — the DESTRUCTIVE CONFIRM, in pieces, and in Go rather than in the template.
//
// It lives here for the same reason the R-353 outcome strings do: this sentence is the only thing
// standing between a customer and the loss of everything they made since the copy was taken, and a
// silent edit to it is how a warning drifts into a reassurance. Named constants can be asserted
// verbatim by a test; a sentence assembled inside an HTML attribute cannot, and R-364 makes
// grepping accented Hungarian out of rendered markup an unreliable check on top of that.
//
// The three parts are three separate obligations:
// Base — what this action DOES: it overwrites, including the database and internal volumes.
// Date — WHICH copy it overwrites with. Two forms, because a stamp that only records an
// ATTEMPT must not be presented as the date of a copy (R-101).
// Contrast — how it differs from the additive button beside it. The register's own requirement:
// a destructive operation reached from a non-destructive surface must carry the
// difference in the confirm, not rely on the customer inferring it from a label.
tier2UnitActionLabel = "Teljes visszaállítás a másolatból"
tier2UnitConfirmBase = "Ez a művelet FELÜLÍRJA az alkalmazás jelenlegi adatait – az adatbázisát és a belső köteteit is – a második meghajtón lévő másolattal. Ami a másolat óta keletkezett, elveszik."
tier2UnitConfirmDateFmt = " A másolat kelte: %s."
tier2UnitConfirmDateUnprovenFmt = " A másolat kelte: %s – ez az utolsó mentési kísérlet ideje, azt nem tudjuk igazolni, hogy sikeres volt."
tier2UnitConfirmContrast = " A mellette lévő „Fájlok visszaállítása” ezzel szemben csak a hiányzó fájlokat pótolja, és semmit nem ír felül. Az alkalmazás a művelet idejére leáll."
)
// tier2UnitConfirmMsg assembles the destructive confirm for one app's Tier-2 unit restore. Pure, so
// the wording is unit-testable; the date is rendered by the SAME helper the rest of the surface uses.
//
// A copy with no recorded date at all still gets a confirm — it just cannot name one. Dropping the
// whole confirm because a date is missing would remove the warning and keep the destruction.
func tier2UnitConfirmMsg(copyDate string, proven bool) string {
msg := tier2UnitConfirmBase
if copyDate != "" {
if proven {
msg += fmt.Sprintf(tier2UnitConfirmDateFmt, fmtRFC3339Local(copyDate))
} else {
msg += fmt.Sprintf(tier2UnitConfirmDateUnprovenFmt, fmtRFC3339Local(copyDate))
}
}
return msg + tier2UnitConfirmContrast
}
// tier2UnitSourceMsg renders the "which copy" clause for the Tier-2 unit restore's outcome, or "" if
// no date is recorded at all. It asks Tier2CopyDate — the SAME resolver the surface uses to pick the
// date it puts in the confirm — so the sentence the customer approves and the sentence they are left
// with cannot name different copies.
func tier2UnitSourceMsg(cov backup.Tier2Coverage) string {
date, proven := cov.Tier2CopyDate()
if date == "" {
return ""
}
if proven {
return fmt.Sprintf(tier2UnitRestoreSourceMsgFmt, fmtRFC3339Local(date))
}
return fmt.Sprintf(tier2UnitRestoreSourceUnprovenMsgFmt, fmtRFC3339Local(date))
}
// backupTier2RestoreHandler (C2, closes F2) restores an app's MISSING user files in place from its
// recorded Tier-2 copy — additive-only: existing live files are never overwritten and nothing is
// ever deleted (see backup.RestoreTier2Files). Same handler shape as backupRestoreHandler.
@@ -1626,10 +1737,19 @@ func (s *Server) backupTier2RestoreHandler(w http.ResponseWriter, r *http.Reques
// about data the restore never examined, at the exact moment they pressed it BECAUSE data was
// missing. Only the no-coverage case is pre-flighted; every other refusal keeps its existing async
// path so this change cannot alter behaviour anywhere else.
//
// R-103 SPLIT THE REFUSAL IN TWO. „no files to restore" has two different answers now, and giving
// both customers the same sentence is what sent one of them to another page for data that is
// restorable on this one.
cov, covErr := s.backupMgr.Tier2RestoreCoverage(stackName)
if covErr == nil && !cov.CanRestore() {
s.logger.Printf("[WARN] [web] Tier-2 file restore refused up front: stack=%s has no restorable subtree in its copy (unit_present=%v) — app NOT stopped", stackName, cov.HasUnit)
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(tier2NoCoverageMsg), http.StatusFound)
msg := tier2NoCoverageMsg
if cov.CanRestoreUnit() {
msg = tier2UnitAvailableMsg
}
s.logger.Printf("[WARN] [web] Tier-2 file restore refused up front: stack=%s has no restorable subtree in its copy (unit_present=%v unit_restorable=%v) — app NOT stopped",
stackName, cov.HasUnit, cov.CanRestoreUnit())
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
return
}
@@ -1667,6 +1787,84 @@ func (s *Server) backupTier2RestoreHandler(w http.ResponseWriter, r *http.Reques
http.Redirect(w, r, "/backups/apps?flash="+url.QueryEscape("Fájl-visszaállítás elindult — az állapot itt frissül."), http.StatusFound)
}
// backupTier2UnitRestoreHandler (R-102/R-103) restores an app IN FULL from the recovery unit mirrored
// onto the SECOND DRIVE — the app's database dumps and named-volume tars, not just its loose files.
//
// It is the destructive twin of backupTier2RestoreHandler above and shares its shape deliberately:
// same guards, same single-writer refusal, same async goroutine, same status banner. What it does not
// share is its promise. The file restore only ever ADDS what is missing; this one OVERWRITES the
// app's live data with the copy's. The template's confirm carries that difference in words, and the
// two actions stay two buttons for exactly that reason (§8: they are different promises).
//
// The pre-flight refusal is the fail-closed gate: a `recovery-unit/` directory that exists is not a
// package. Refusing here means the app is never stopped for a mirror that could not have been read.
func (s *Server) backupTier2UnitRestoreHandler(w http.ResponseWriter, r *http.Request) {
_ = r.ParseForm()
stackName := r.FormValue("stack_name")
if stackName == "" {
http.Redirect(w, r, "/backups/apps?flash_error=Hi%C3%A1nyz%C3%B3+param%C3%A9terek", http.StatusFound)
return
}
// Same F2-defense as both restores beside it: a stack name is a single segment, never a path.
if !validStackName(stackName) {
s.logger.Printf("[WARN] [web] Tier-2 unit restore rejected: invalid stack_name %q from %s", stackName, r.RemoteAddr)
http.Redirect(w, r, "/backups/apps?flash_error=%C3%89rv%C3%A9nytelen+alkalmaz%C3%A1sn%C3%A9v", http.StatusFound)
return
}
if s.backupMgr == nil {
http.Redirect(w, r, "/backups/apps?flash_error=Ment%C3%A9s+nincs+be%C3%A1ll%C3%ADtva", http.StatusFound)
return
}
// R-351b (Scenario H): a second press — by button or by a direct POST — must not start a second
// run. restoreOpBlocked() and not IsRunning(), because the concurrency flag is only taken inside
// the goroutine, after this handler has already returned.
if msg, blocked := s.restoreOpBlocked(); blocked {
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
return
}
// Pre-flight, before any op is begun and before the app is stopped: does this copy hold a unit
// this restore can actually open? Only the no-unit case is pre-flighted; every other refusal keeps
// its existing async path, so this cannot alter behaviour anywhere else.
cov, covErr := s.backupMgr.Tier2RestoreCoverage(stackName)
if covErr != nil {
s.logger.Printf("[WARN] [web] Tier-2 unit restore refused up front: stack=%s: %v — app NOT stopped", stackName, covErr)
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(covErr.Error()), http.StatusFound)
return
}
if !cov.CanRestoreUnit() {
s.logger.Printf("[WARN] [web] Tier-2 unit restore refused up front: stack=%s has no openable unit in its copy (unit_present=%v) — app NOT stopped", stackName, cov.HasUnit)
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(tier2NoCoverageMsg), http.StatusFound)
return
}
s.logger.Printf("[WARN] [web] Tier-2 UNIT restore requested (async, OVERWRITES live data): stack=%s from %s", stackName, r.RemoteAddr)
s.backupMgr.BeginRestoreOp("tier2-unit-restore", stackName)
go func() {
start := time.Now()
res, err := s.backupMgr.RestoreTier2Unit(stackName)
if err != nil {
s.logger.Printf("[ERROR] [web] Tier-2 unit restore failed (async): stack=%s: %v", stackName, err)
s.backupMgr.EndRestoreOp(false, "Teljes visszaállítás sikertelen: "+err.Error())
return
}
s.logger.Printf("[INFO] [web] Tier-2 unit restore completed (async): stack=%s in %s (volumes %d/%d, dbs %d/%d)",
stackName, time.Since(start), res.VolumesReplayed, res.ManifestVolumes, res.DBsReplayed, res.ManifestDBs)
// The outcome sentence is yesterday's — unitRestoreOutcomeMsg, unchanged, because what came
// back is the same fact whichever unit it came out of, and a second wording of it would be a
// second thing to keep honest. What IS added is which copy it came from and how old that copy
// is: this action overwrote the customer's live data, and the sentence they are left with has
// to say what it overwrote it with (Scenario E).
msg := unitRestoreOutcomeMsg(stackName, res)
if src := tier2UnitSourceMsg(cov); src != "" {
msg += " " + src
}
s.backupMgr.EndRestoreOp(true, msg)
}()
http.Redirect(w, r, "/backups/apps?flash="+url.QueryEscape("Teljes visszaállítás elindult — az állapot itt frissül."), http.StatusFound)
}
// settingsBaseData is the shared identity block used by every settings-family subpage
// (D1 split: /settings, /settings/notifications, /settings/security, /storage).
func (s *Server) settingsBaseData(page, title string) map[string]interface{} {
@@ -0,0 +1,413 @@
package web
import (
"io"
"log"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-103 Group D — the surface: the refusal becomes an action.
//
// Before v0.229.0 an app whose Tier-2 copy held only a recovery unit got a message telling it to
// press a button on a DIFFERENT page. The data it was asking for is in the copy it is looking at, and
// since R-102 it is restorable from there. The action now lives where the refusal was.
//
// Every string assertion here compares against the NAMED CONSTANT rather than a Hungarian literal
// retyped in the test. That is R-364 discipline in its strongest form: a grep for accented text
// returned zero for strings that were present, and a re-typed literal can differ from the shipped one
// by a character nobody sees.
// --- the surface (template) -----------------------------------------------------------------
// r103Row is a Tier-2-configured row with a successful copy — the shape the template needs before it
// will render the actions block at all.
func r103Row(unitRestorable bool, date string, proven bool) AppBackupRow {
row := AppBackupRow{
StackName: "docmost", DisplayName: "Docmost",
Tier2Configured: true, Tier2Dest: "flash", Tier2Schedule: "Naponta",
Tier2LastRun: date, Tier2LastStatus: "ok", Tier2StatusBadge: "Sikeres",
Tier2LastSuccess: date, Tier2SuccessTracked: true,
Tier2UnitRestorable: unitRestorable,
}
if unitRestorable {
row.Tier2CopyDate, row.Tier2CopyDateProven = date, proven
row.Tier2UnitConfirm = tier2UnitConfirmMsg(date, proven)
}
return row
}
// D1 — TestR103_UnitActionOfferedWhenTheMirrorExists.
func TestR103_UnitActionOfferedWhenTheMirrorExists(t *testing.T) {
html := renderBackupPage(t, "backups_apps", baseBackupData([]AppBackupRow{
r103Row(true, "2026-08-25T03:30:00Z", true),
}))
if !strings.Contains(html, `action="/backup/tier2/unit-restore"`) {
t.Error("the unit-restore form is not on the page — the refusal is still a dead end")
}
if !strings.Contains(html, tier2UnitActionLabel) {
t.Errorf("the action is not labelled %q", tier2UnitActionLabel)
}
// The additive action must still be there, separately. Merging them is forbidden: they are
// different promises (one adds, one overwrites).
if !strings.Contains(html, `action="/backup/tier2/restore"`) {
t.Error("the additive file restore disappeared from the row")
}
// The destructive one is visually distinct from the additive one beside it.
if !strings.Contains(html, "btn-danger-outline") {
t.Error("the destructive action does not carry a danger style")
}
}
// D2 — TestR103_UnitActionAbsentWhenItDoesNot. Scenario G: no legs and no openable unit → an honest
// refusal, and NO unit action. A button offered over a copy the restore would refuse is worse than no
// button, because it is a promise withdrawn at the moment of use.
func TestR103_UnitActionAbsentWhenItDoesNot(t *testing.T) {
html := renderBackupPage(t, "backups_apps", baseBackupData([]AppBackupRow{
r103Row(false, "2026-08-25T03:30:00Z", true),
}))
if strings.Contains(html, "/backup/tier2/unit-restore") {
t.Error("the unit action was offered for a copy with no openable unit")
}
if strings.Contains(html, tier2UnitActionLabel) {
t.Error("the unit action's label leaked onto a row that must not offer it")
}
// NEGATIVE CONTROL for D1's assertions: the row itself did render, so D1's positives were not
// an artefact of the whole block being absent.
if !strings.Contains(html, `action="/backup/tier2/restore"`) {
t.Fatal("the row did not render at all — D1's positive assertions prove nothing")
}
}
// D3 — TestR103_ConfirmStatesTheOverwrite. The confirm must carry the DIFFERENCE the register
// requires: a destructive operation reached from a non-destructive surface says so, and says how it
// differs from the action beside it.
func TestR103_ConfirmStatesTheOverwrite(t *testing.T) {
confirm := tier2UnitConfirmMsg("2026-08-25T03:30:00Z", true)
if !strings.Contains(confirm, tier2UnitConfirmBase) {
t.Error("the confirm does not state that the operation OVERWRITES live data")
}
if !strings.Contains(confirm, tier2UnitConfirmContrast) {
t.Error("the confirm does not state how it differs from the additive restore beside it")
}
// NEGATIVE CONTROL: the additive restore's own confirm must NOT have acquired this language.
// Without it, a test that passed because both buttons warn about overwriting would look green.
html := renderBackupPage(t, "backups_apps", baseBackupData([]AppBackupRow{
r103Row(true, "2026-08-25T03:30:00Z", true),
}))
fileConfirmStart := strings.Index(html, "Visszaállítja a hiányzó fájlokat")
if fileConfirmStart < 0 {
t.Fatal("the additive confirm is gone — the negative control has nothing to check")
}
fileConfirm := html[fileConfirmStart:]
if end := strings.Index(fileConfirm, `">`); end > 0 {
fileConfirm = fileConfirm[:end]
}
if strings.Contains(fileConfirm, "FEL") {
t.Errorf("the ADDITIVE confirm gained overwrite language: %q", fileConfirm)
}
// And the confirm reaches the markup as the rendered attribute, not only as a Go constant.
if !strings.Contains(html, `data-confirm=`) {
t.Error("no data-confirm attribute rendered")
}
if !strings.Contains(html, "FEL") {
t.Error("the destructive wording never reached the page")
}
}
// D4 — TestR103_ConfirmNamesTheCopyDate. Scenario E. The action overwrites live data with a copy of a
// certain age, and „nobody restores last week over today by accident" is only true if the date is on
// the screen. R-101 governs the wording when the date is an ATTEMPT rather than a proven copy.
func TestR103_ConfirmNamesTheCopyDate(t *testing.T) {
const stamp = "2026-08-25T03:30:00Z"
rendered := fmtRFC3339Local(stamp)
if rendered == stamp {
t.Fatal("the stamp did not render as a local date — the assertion below would be vacuous")
}
proven := tier2UnitConfirmMsg(stamp, true)
if !strings.Contains(proven, rendered) {
t.Errorf("the confirm does not name the copy's date: %q", proven)
}
unproven := tier2UnitConfirmMsg(stamp, false)
if !strings.Contains(unproven, rendered) {
t.Errorf("the unproven confirm does not name the date: %q", unproven)
}
if proven == unproven {
t.Error("a proven copy and a bare attempt got the SAME sentence — R-101 exactly")
}
// A copy with no recorded date still gets the warning; it just cannot name one.
none := tier2UnitConfirmMsg("", false)
if !strings.Contains(none, tier2UnitConfirmBase) || !strings.Contains(none, tier2UnitConfirmContrast) {
t.Errorf("a dateless copy lost its confirm entirely: %q", none)
}
// And it is on the page, not merely constructible.
html := renderBackupPage(t, "backups_apps", baseBackupData([]AppBackupRow{r103Row(true, stamp, true)}))
if !strings.Contains(html, rendered) {
t.Errorf("the copy's date %q is not in the rendered row", rendered)
}
}
// TestR103_AvailableMsgNamesTheButtonByItsLabel — the refusal that now points AT the new action must
// name it by the label the button actually carries, or it points at nothing.
func TestR103_AvailableMsgNamesTheButtonByItsLabel(t *testing.T) {
if !strings.Contains(tier2UnitAvailableMsg, tier2UnitActionLabel) {
t.Errorf("tier2UnitAvailableMsg does not name %q", tier2UnitActionLabel)
}
// It must NOT send anyone to another page any more; that is the R-103 defect it replaces.
if strings.Contains(tier2UnitAvailableMsg, "oldalon") {
t.Error("the message still routes the customer to another page for a copy restorable here")
}
// The surviving no-coverage message still names the route that works.
if !strings.Contains(tier2NoCoverageMsg, "oldalon") {
t.Error("tier2NoCoverageMsg no longer names any route — Scenario G requires one")
}
// C3 — tier2UnitNotCoveredMsg is NOT deleted: it is still appended where the FILE restore ran.
if tier2UnitNotCoveredMsg == "" {
t.Fatal("tier2UnitNotCoveredMsg was deleted")
}
if !strings.Contains(readSourceFile(t, "handlers.go"), `msg += " " + tier2UnitNotCoveredMsg`) {
t.Error("tier2UnitNotCoveredMsg is no longer appended by the file-restore handler")
}
}
func readSourceFile(t *testing.T, name string) string {
t.Helper()
b, err := os.ReadFile(name)
if err != nil {
t.Fatal(err)
}
return string(b)
}
// --- the handler ----------------------------------------------------------------------------
// r103Provider is a StackDataProvider that completes every lifecycle call, so the restore goroutine
// runs to its outcome instead of parking.
type r103Provider struct{ hdd string }
func (p *r103Provider) GetStackComposePath(string) (string, bool) { return "", false }
func (p *r103Provider) ListDeployedStacks() []backup.StackSummary { return nil }
func (p *r103Provider) GetStackHDDMounts(string) []string { return nil }
func (p *r103Provider) GetStackHDDPath(string) string { return p.hdd }
func (p *r103Provider) GetImportRoot() string { return "" }
func (p *r103Provider) GetDockerVolumes(string) []string { return nil }
func (p *r103Provider) StopStack(string) error { return nil }
func (p *r103Provider) StartStack(string) error { return nil }
func (p *r103Provider) RefreshAndIsRunning(string) bool { return true }
func (p *r103Provider) GetStackRecoveryInfo(string) (backup.RecoveryInfo, bool) {
return backup.RecoveryInfo{}, false
}
func (p *r103Provider) GetStackClassifiedBinds(string) ([]backup.ClassifiedBind, bool) {
return nil, false
}
func (p *r103Provider) RecoverStackSecrets(string, []string) map[string]string { return nil }
func (p *r103Provider) RecreateStackDefinitionFromUnit(string, string, map[string]string) error {
return nil
}
func (p *r103Provider) StartStackServices(string, []string) error { return nil }
const r103CopyStamp = "2026-08-25T03:30:00Z"
// newR103Server wires a Server over a real backup.Manager whose recorded Tier-2 copy for "app" holds
// an OPENABLE recovery unit mirror. Nothing is stubbed between the handler and the manager: the whole
// point of D6 is that the wiring is what is under test.
func newR103Server(t *testing.T, withUnit bool) (*Server, *backup.Manager) {
t.Helper()
tmp := t.TempDir()
live := filepath.Join(tmp, "usb")
dest := filepath.Join(tmp, "flash")
lg := log.New(io.Discard, "", 0)
sett, err := settings.Load(filepath.Join(tmp, "settings.json"), lg)
if err != nil {
t.Fatal(err)
}
for _, p := range []string{live, dest} {
if err := sett.AddStoragePath(settings.StoragePath{Path: p, Label: filepath.Base(p)}); err != nil {
t.Fatal(err)
}
}
if err := sett.SetCrossDriveConfig("app", &settings.CrossDriveBackup{
Enabled: true, Method: "rsync", DestinationPath: dest,
LastRun: r103CopyStamp, LastSuccess: r103CopyStamp, SuccessTracked: true, LastStatus: "ok",
}); err != nil {
t.Fatal(err)
}
destBase := filepath.Join(dest, "backups", "secondary", "app")
write := func(rel, body string) {
p := filepath.Join(destBase, rel)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
write(".felhom-tier2-layout", "2")
if withUnit {
write("recovery-unit/manifest.json", `{"schema_version":1,"app_name":"app"}`)
write("recovery-unit/compose/app.yaml", "deployed: true\nenv:\n SUBDOMAIN: app\n")
write("recovery-unit/compose/docker-compose.yml", "services:\n app:\n image: example/app:1\n")
} else {
// A directory that exists and is not a package — the fail-closed case.
if err := os.MkdirAll(filepath.Join(destBase, "recovery-unit"), 0o755); err != nil {
t.Fatal(err)
}
}
cfg := &config.Config{}
cfg.Paths.DataDir = tmp
m := backup.NewManager(cfg, sett, lg)
m.SetStackProvider(&r103Provider{hdd: live})
return &Server{cfg: cfg, backupMgr: m, logger: lg}, m
}
func postTier2UnitRestore(t *testing.T, s *Server, stack string) *httptest.ResponseRecorder {
t.Helper()
form := url.Values{"stack_name": {stack}}
req := httptest.NewRequest(http.MethodPost, "/backup/tier2/unit-restore", strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
rec := httptest.NewRecorder()
s.backupTier2UnitRestoreHandler(rec, req)
return rec
}
func waitRestoreDone(t *testing.T, m *backup.Manager) backup.RestoreOpStatus {
t.Helper()
deadline := time.Now().Add(30 * time.Second)
for time.Now().Before(deadline) {
st := m.RestoreStatus()
if !st.Running && st.Last != nil {
return st
}
time.Sleep(50 * time.Millisecond)
}
t.Fatal("the restore never published a result")
return backup.RestoreOpStatus{}
}
// D5 — TestR103_SecondPressIsRefused. Scenario H, R-351b. The concurrency flag is only taken inside
// the goroutine, AFTER the handler returns, so a guard reading IsRunning() alone leaves a window in
// which a second press starts a second run and is told „elindult".
func TestR103_SecondPressIsRefused(t *testing.T) {
s, m := newR103Server(t, true)
m.BeginRestoreOp("restore", "other-app") // a restore is already in flight, display-flag set
rec := postTier2UnitRestore(t, s, "app")
loc := rec.Header().Get("Location")
if !strings.Contains(loc, "flash_error") {
t.Fatalf("a second press was accepted: %q", loc)
}
// The identity of the FIRST operation survives — the consequence, not which flag was read.
if st := m.RestoreStatus(); st.Stack != "other-app" {
t.Errorf("the in-flight op was replaced by the refused one: %+v", st)
}
}
// D6 — TestR103_HandlerPublishesTheOutcome. THE SEAM TEST.
//
// It reads the outcome off RestoreStatus().Last.Message — the only place a customer sees it — rather
// than calling the manager and inspecting a return value. Three shipped defects in this project came
// from testing a component whose caller never invoked it (R-106's fakeObserver being the clearest),
// and the mechanism here is exactly that shape: a correct RestoreTier2Unit wired to nothing would
// leave the banner silent and every manager-level test green.
//
// Red-proof (recorded in REPORT.md): drop the `s.backupMgr.EndRestoreOp(true, msg)` call at the end
// of the handler's goroutine → this test fails on "the restore never published a result".
func TestR103_HandlerPublishesTheOutcome(t *testing.T) {
s, m := newR103Server(t, true)
rec := postTier2UnitRestore(t, s, "app")
if rec.Code != http.StatusFound {
t.Fatalf("status = %d, want 302", rec.Code)
}
if loc := rec.Header().Get("Location"); strings.Contains(loc, "flash_error") {
t.Fatalf("the restore was refused: %q", loc)
}
st := waitRestoreDone(t, m)
if !st.Last.OK {
t.Fatalf("outcome reported failure: %q", st.Last.Message)
}
if st.Last.Stack != "app" || st.Last.Op != "tier2-unit-restore" {
t.Errorf("the published result names the wrong operation: op=%q stack=%q", st.Last.Op, st.Last.Stack)
}
// The sentence is yesterday's honest outcome PLUS the clause naming which copy overwrote the live
// data (Scenario E). Asserted as an exact composition so neither half can silently vanish.
wantOutcome := unitRestoreOutcomeMsg("app", backup.UnitRestoreResult{})
wantSource := tier2UnitSourceMsg(backup.Tier2Coverage{CopyLastSuccess: r103CopyStamp})
if wantSource == "" {
t.Fatal("the fixture recorded no copy date — the assertion below would be vacuous")
}
if want := wantOutcome + " " + wantSource; st.Last.Message != want {
t.Errorf("published message =\n %q\nwant\n %q", st.Last.Message, want)
}
if !strings.Contains(st.Last.Message, fmtRFC3339Local(r103CopyStamp)) {
t.Error("the outcome does not name the date of the copy it restored from")
}
}
// TestR103_UnitRestoreRefusesAnUnopenableMirrorWithoutStopping — the handler's fail-closed pre-flight.
// A directory is not a package, and refusing before BeginRestoreOp means no banner, no outage and no
// rewritten definition.
func TestR103_UnitRestoreRefusesAnUnopenableMirrorWithoutStopping(t *testing.T) {
s, m := newR103Server(t, false)
rec := postTier2UnitRestore(t, s, "app")
loc := rec.Header().Get("Location")
if !strings.Contains(loc, "flash_error") {
t.Fatalf("the restore was accepted over an unopenable mirror: %q", loc)
}
if !strings.Contains(loc, url.QueryEscape(tier2NoCoverageMsg)) {
t.Errorf("refusal flash = %q, want tier2NoCoverageMsg", loc)
}
if st := m.RestoreStatus(); st.Running || st.Last != nil {
t.Errorf("an operation was begun despite the refusal: %+v", st)
}
}
// TestR103_UnitRestoreHandlerGuards — the same three guards the two restores beside it carry, proven
// to run BEFORE any work (backupMgr is nil, so reaching it would panic).
func TestR103_UnitRestoreHandlerGuards(t *testing.T) {
s := &Server{logger: log.New(io.Discard, "", 0)} // backupMgr nil on purpose
for name, want := range map[string]string{
"../../etc": "%C3%89rv%C3%A9nytelen+alkalmaz%C3%A1sn%C3%A9v",
"a/b": "%C3%89rv%C3%A9nytelen+alkalmaz%C3%A1sn%C3%A9v",
"": "Hi%C3%A1nyz%C3%B3+param%C3%A9terek",
} {
rec := postTier2UnitRestore(t, s, name)
if loc := rec.Header().Get("Location"); !strings.Contains(loc, want) {
t.Errorf("%q: redirect = %q, want flash %q", name, loc, want)
}
}
rec := postTier2UnitRestore(t, s, "docmost")
if loc := rec.Header().Get("Location"); !strings.Contains(loc, "Ment%C3%A9s+nincs+be%C3%A1ll%C3%ADtva") {
t.Errorf("nil backupMgr: redirect = %q", loc)
}
}
// TestR103_FileRestoreRefusalPointsAtTheActionThatWorks — the R-103 split. An app with no file legs
// but a restorable unit gets the message that names the button beside it, NOT the one that sends it
// to another page.
func TestR103_FileRestoreRefusalPointsAtTheActionThatWorks(t *testing.T) {
s, _ := newR103Server(t, true)
rec := postTier2Restore(t, s, "app")
loc := rec.Header().Get("Location")
if !strings.Contains(loc, url.QueryEscape(tier2UnitAvailableMsg)) {
t.Errorf("refusal flash = %q, want tier2UnitAvailableMsg", loc)
}
if strings.Contains(loc, url.QueryEscape(tier2NoCoverageMsg)) {
t.Error("the old dead-end message is still shown for a copy restorable here")
}
}
+4
View File
@@ -537,6 +537,10 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// C2: in-place, additive-only file restore from the Tier-2 copy (class-C user files)
case path == "/backup/tier2/restore" && r.Method == http.MethodPost:
s.backupTier2RestoreHandler(w, r)
// R-102/R-103: the DESTRUCTIVE twin — a full recovery-unit restore read from the SECOND DRIVE's
// mirror. Separate route because it is a separate promise: this one overwrites live data.
case path == "/backup/tier2/unit-restore" && r.Method == http.MethodPost:
s.backupTier2UnitRestoreHandler(w, r)
// Off-box (NAS) restic-SFTP backup (Part B)
case path == "/backup/offbox/config" && r.Method == http.MethodPost:
s.offboxConfigHandler(w, r)
@@ -261,6 +261,17 @@
{{else}}
<span class="layer-reason" style="opacity:.85">Még nincs sikeres másolat, amiből vissza lehetne állítani.</span>
{{end}}
{{/* R-102/R-103: the DESTRUCTIVE twin of the button above. Offered only when the
copy holds an openable recovery unit (Tier2UnitRestorable — a second
predicate, never a widening of the file restore's). It is deliberately a
SEPARATE button in a danger style: the one beside it only adds what is
missing, this one overwrites the app's live data with the copy's. */}}
{{if .Tier2UnitRestorable}}
<form method="POST" action="/backup/tier2/unit-restore" style="display:inline">{{$.CSRFField}}
<input type="hidden" name="stack_name" value="{{.StackName}}">
<button type="submit" class="btn btn-xs btn-danger-outline" data-confirm="{{.Tier2UnitConfirm}}">Teljes visszaállítás a másolatból</button>
</form>
{{end}}
<a href="/stacks/{{.StackName}}/backup" class="btn btn-xs btn-outline">Beállítás</a>
</div>
{{else}}