diff --git a/CHANGELOG.md b/CHANGELOG.md index a872cff..051fa36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,50 @@ ## Changelog +### v0.186.0 — R-114 + R-112: tell the truth about the backup target, then show it (2026-07-29) — MinAgent 0.113.0 (unchanged) + +Two defects E-2d found on a real box, fixed in this order deliberately: the message is corrected +BEFORE it is put on screen, because switching on a banner that lies is worse than a silent one. + +**R-114 — the third state.** `resolveBackupTargetState` had two outcomes: a disk claims the target +(healthy), or nothing does (degraded, "the backup is on the system disk"). The state *configured, and +its drive is gone* has no branch, so it fell into the second and inherited both its message and its +offer. Live payload, target detached: `degraded:true, target:"felhom-backup"` **plus** the system-disk +copy — false, the backup was on a drive that had vanished — **plus** `offer_path` naming that same +vanished drive as the remedy (`felhom.eu` `audits/E2D-fresh-vm-2026-07-29.md` §5.3). + +New `BackupTargetState.TargetAbsent` discriminates. `Degraded` keeps its meaning ("is there a +problem") so the wire contract is unchanged for every consumer; `TargetAbsent` answers "which +problem", because the two have OPPOSITE remedies — attach any second drive, versus reconnect *that* +one. Its copy is routed through `degradedMessageFor`, so there is still exactly one place that +decides what a customer reads. **No offer in this state**, suppressed on the branch itself rather +than left to `firstOfferableDrive`'s `Disconnected` skip: that flag is set by the agent-side gate in +another repo (R-113), and this state must be correct independently of it. Belt here, braces there. + +The absent copy is **verbatim** the hub's `customerMessages["backup_target_absent"]`, so the banner +and the email tell one story. It now lives in two repos with nothing binding them but a test — +filed as a drift risk, not solved. + +**R-112 — the state finally has a consumer.** The endpoint was byte-correct and **nothing in the +product ever asked for it**: templates fetch 18 distinct `/api/storage/*` endpoints, and +`backup-target[/assign]` were the only two with zero references. Server-rendered on the backups page +now (`backupsHandler` → `backupTargetView` → `backups.html`), following the existing +`SingleCopyWarning` banner pattern — not a 19th JS fetch, because a banner that needs JavaScript to +appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy +and unknown, so those render nothing at all: no badge, no reassurance. The offer control POSTs to the +existing assign endpoint behind the standard inline confirm, never auto-submits, and surfaces +`restart_required` honestly instead of adding a self-restart. + +**Seam test (Scenario E)** drives `backupsHandler` over httptest and asserts the RENDERED HTML — +handler → view → resolver → template. Deleting the one line that sets `data["BackupTarget"]` +reproduces the R-112 state and fails every render assertion. Tests 326 → 338 (+12) in +`internal/web`; three red-proofs run and reverted. + +**MinAgent unchanged at 0.113.0.** R-114 reads `BackupTarget`/`MountPath`/`GuestPath`/`Role`, none of +which R-113 altered — it changed `BoundUnderParent`, which this code does not read. So demo-hp +(agent 0.113.0) is not held. + +**NOT LIVE-VALIDATED.** Scenario C cannot occur on a healthy box; Session C proves it. + ### v0.185.1 — E-2: the offer endpoints were mounted where nothing routed to them (2026-07-29) Registered as `/api/backup-target` inside `ServeStorageAPI`'s switch — which `main.go` mounts ONLY at diff --git a/REPORT.md b/REPORT.md index dbb57ae..7734fa6 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,84 +1,96 @@ -# REPORT — R-101 + F-DIAG: the customer must not be told a failed backup is a copy (v0.182.0) (2026-07-28) +# REPORT — R-114 + R-112: tell the truth about the backup target, then show it (v0.186.0, 2026-07-29) -**Overwritten** per the standing rule. Controller **v0.181.0 → v0.182.0**. Full arc incl. Phase 0, the -F-OPS runbook and the rendered dialogs: `felhom.eu/REPORT-r101.md`. +**Overwritten** per the standing rule. Controller **v0.185.1 → v0.186.0**. Session B of three +(A → R-113 agent v0.114.0 shipped-unvalidated · **B → this** · C → one drill rebuild proving all three). +`felhom-agent` untouched. **NOTHING HERE IS LIVE-VALIDATED** — see the bottom. -## Baselines (reconfirmed, not copied) -`felhom-controller 3db8bfb` = origin/main, clean; 0.181.0 live on both boxes; hub 0.80.0 ready 1/1. +## Baselines -## R-101 — misinformation at a decision point, not an alarm bug +felhom-controller `cdaeb369726b` v0.185.1 · felhom-agent `b58d7bcf39a9` v0.114.0 (untouched) · +felhom.eu `338b2ccf86`. All clean and pushed at start. -`Tier2LastRun` is the ATTEMPT clock (`recordTier2Failure` writes it too) and it was rendered as -„Legutóbbi másolat" in the **restore confirm dialog**. The restore it guards fills in MISSING files -without touching existing ones — so a customer whose Tier-2 had been failing was told a copy existed -from last night, restored, and silently received **older** files while believing they were recent. No -error, no signal, and the customer's own reason for restoring quietly defeated. +**MinAgent decision: unchanged at 0.113.0.** R-114 reads `BackupTarget`, `MountPath`, `GuestPath`, +`Role` — **none of which R-113 altered**; it changed `BoundUnderParent`, which this code does not +read (`internal/agentapi/client.go:303-338`). Consequence, stated rather than discovered: **demo-hp, +still on agent 0.113.0, is not held** and needs no move for this release. -`CrossDriveBackup` gains `LastSuccess` (the offsite anchor's rule and shape, not a second -implementation) plus `SuccessTracked`, which distinguishes "predates the anchor" from "has one and it -is empty". That marker is not optional: **all 7 Tier-2 rows on the fleet** were pre-anchor at deploy, -so without it every customer would have been shown „Még nincs sikeres másolat" at once. Legacy rows -migrate truthfully on first touch — an `ok` row adopts its time, an `error` row seeds nothing. +## Part 1 — R-114: the third state -Three dishonest render sites fixed; the main configured branch and the shares surface were already -honest. The dialog also stops printing raw UTC RFC3339 — new `fmtTimeStr` gives Budapest-local -`2026-07-28 16:40`. +`resolveBackupTargetState` had two outcomes — a disk claims the target (healthy), or nothing does +(degraded, "the backup is on the system disk"). *Configured, and its drive is gone* had no branch, so +it fell into the second and inherited its message **and** its offer. Observed live: +`degraded:true, target:"felhom-backup"` + the system-disk copy (false) + `offer_path` naming the drive +that had just vanished (`audits/E2D-fresh-vm-2026-07-29.md` §5.3). -## Part 2 — the copy-site hazard, in the path of Part 1 +- New `BackupTargetState.TargetAbsent`. `Degraded` keeps its meaning — *is there a problem* — so the + wire contract is unchanged for every consumer; `TargetAbsent` answers *which* problem, because the + two have opposite remedies. +- Discriminator `targetIsConfiguredDrive`: a real storage id, versus the builtin `local` or an unset + tier (both = never configured). +- Copy routed through `degradedMessageFor`, so one place still decides what a customer reads. +- **Offer suppressed on the branch itself**, not left to `firstOfferableDrive`'s `Disconnected` skip — + that flag comes from the agent-side gate in another repo, and this state must be right without it. -The three `record*` helpers each built a WHOLE struct literal with two fields re-applied; everything -else was zeroed on every status write. Adding `LastSuccess` to that shape would have had -`recordTier2Failure` **clear** it — the mirror image of the defect, firing on the first failure. -`tier2Update` now copies the existing row and overlays the outcome: **safe by construction**. Sweep: -`SetTier2Preference` mutates in place (safe), `SetCrossDriveConfig(name, nil)` is a deliberate delete. +## Part 2 — R-112: the state gets a consumer -## F-DIAG — six classes, and a leak I caught in my own fix +The endpoint was byte-correct and **nothing asked for it** — templates fetch 18 distinct +`/api/storage/*` endpoints and these were the only two with zero references (§5.1). -`ClassifyOffsiteFailure` → `quota` / `orphaned` / `no_repo` / `no_units` / `transport` / **`unknown`**, -each with its own Hungarian message; unclassifiable says so rather than being folded into a neighbour. +Server-rendered on `/backups`, following the existing `SingleCopyWarning` banner at `backups.html:11`. +`backupsHandler` → `backupTargetView` → template. **Not a 19th JS fetch**: the state is already +resolved when the page is built, and a banner needing JavaScript to appear is one more thing that can +silently not happen. `backupTargetView` returns **nil** for healthy and unknown, so a template typo +cannot decorate a working box. The offer control POSTs to the existing assign endpoint behind the +standard inline confirm, never auto-submits, and surfaces `restart_required` honestly. -The old message was a raw `err.Error()` passthrough carrying `sftp:@:` off the box. -My first sanitiser regex-matched `sftp:…` and `user@host` and looked complete — **its own test caught it -leaking** on `ssh: connect to host port 23: Connection refused`, a bare hostname in neither -shape. It now redacts the target's **actual** host/user/path literally, regex kept as a backstop. +**Placement:** the E-2 design doc names no surface — `07-backup-architecture.md` has no banner or +degraded reference, and the only `E-2` string in the capability map is the unrelated `CAMPAIGN-6E-2`. +So the choice is mine: the backups page already hosts a same-severity backup warning and the copy is +about `rendszermentés`. -## Red-proofs — all observed failing -| # | red-proof | observed failure | +## Files + +`internal/web/backup_target_offer.go` (third state, view, tiers seam) · `internal/web/server.go` +(`tiersFn`) · `internal/web/handlers.go` (one line wiring the view) · +`internal/web/templates/backups.html` (banner + offer + assign script) · two new test files · +`CHANGELOG.md` · `REUSE.md` · `controller/README.md`. + +## Tests: 326 → 338 (+12) in `internal/web`. Suite green (27 packages), both template gates pass. + +**Scenario E — the seam test, which is the one whose absence let E-2 ship.** It does *not* call the +resolver and assert a string (that proves the resolver, which was never broken). It drives +`backupsHandler` over `httptest` and asserts the **rendered HTML**, travelling +handler → `backupTargetView` → `resolveBackupTargetState` → `degradedMessageFor` → the production +`backups` template. + +| Red-proof | Mutation | Failure text produced | |---|---|---| -| A | dialog back on the attempt clock | `the dialog does not name the last SUCCESSFUL copy` | -| C | gate the restore on `LastRun` | `a tier that has NEVER succeeded still offers a restore` | -| D | caution made unconditional | `a HEALTHY tier shows the failed-attempt caution ("nem sikerült")` | -| F | clear the anchor on failure | `a FAILED run wiped the success anchor (round 1)` | -| + | raw sanitiser | `the repo reference reached the message ("sftp:" leaked)` | +| 1 | delete the `targetIsConfiguredDrive` branch | `TargetAbsent is false for target "felhom-backup"…` **and** `offered "/mnt/felhom-drives/mentes2" while the configured target is absent — E-2d observed this exact payload offering the drive that had just vanished` | +| 2 | drop the absent case from `degradedMessageFor` | `the customer is told the backup is on the SYSTEM DISK, which is false…` + all three copy fragments missing | +| 3 | delete `data["BackupTarget"] = …` from the handler | `the absent-drive copy never reached the page…` + every render assertion in the never-configured case | -F runs the **real** `recordTier2Success` → `recordTier2Failure` sequence; A/C/D **render the production -template tree** and assert the string the customer reads. +All three reverted; files verified byte-identical after. -`go build`, `go vet ./...`, `go test ./...` — 27 packages `rc=0`; template-ID and emoji gates OK. +**Negative assertions:** healthy and unknown are checked for the absence of **all three** copy strings +and of the banner element — not for the absence of an error. -## LIVE on demo-hp — the rendered dialog +**A test caught its own imprecision:** the first Scenario-C assertion matched the bare id +`backup-target-assign`, which also appears inside the page script that always renders, so it reported +a control that was not there. Tightened to the control's own markup. -Failure induced by moving the Tier-2 destination aside and leaving a file in its place, so -`mkdir …/recovery-unit` fails. (`chmod` is useless — the controller is root; `chattr +i` is refused in -an unprivileged container. Both tried, both reported.) Data only moved, never deleted; restored intact. +## NOT LIVE-VALIDATED — awaiting Session C -``` -status=error last_run=14:42:18Z (ADVANCED) last_success=14:40:55Z (HELD) -``` -**failed state:** `… Legutóbbi sikeres másolat: 2026-07-28 16:40. Figyelem: a legutóbbi mentési -kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek lehetnek.` -**healthy state:** `… Legutóbbi sikeres másolat: 2026-07-28 16:43.` — no caution, no tonal change. -**legacy state (pre-run):** `Legutóbbi másolat: 2026-07-28 03:30` — today's wording, unchanged. +- **Scenario C cannot occur on a healthy box.** The demo guest has a present target, so the + absent-drive banner is unproven live by construction. +- Untested live: the offer control's click → assign → `restart_required` surfacing. +- The live check in this session proves only that the seam is wired and which render a healthy box + produces — that is R-112's whole point, but it is not proof of R-114. -Restored: destination a directory again, 86 MB intact, mode 755, `status=ok`. -**demo-felhom is the untouched control** — all 5 rows still legacy, rendering today's way, 15/15 up. +## Observations — filed, not fixed -## Deployed -`felhom-controller` **0.182.0** on demo-hp and demo-felhom, both `Up (healthy)`. - -## NOT yet live-validated (carried forward) -- **F-DIAG's classes** — unit-proven; no live failure of each class was induced. -- **Scenario C live** — never-succeeded is unit-proven only; no fleet row is in that state and - manufacturing one means destroying an app's only Tier-2 history. -- **The Tier-2 restore itself** — this arc changed what the dialog says, not what the restore does. -- R-100's 48h threshold (injected clock), fault 4, R-99, F-HUB, fault 12, the concurrency overlap. +1. **The absent copy now lives in two repos** — here and hub `internal/notify/templates.go:93` — with + nothing binding them but `TestAbsentCopyMatchesTheHubEmailWordForWord`, which only fails if *this* + side drifts. Reword the hub and the test still passes while the banner and the email disagree. +2. **`resolveBackupTargetState` bypassed the existing `disksFn` seam** and called the client directly, + which is why its branching had no test at all before this session. Now routed through `fetchDisks` + with a sibling `tiersFn` — worth checking whether other agent-reading paths do the same. diff --git a/REUSE.md b/REUSE.md index e6284da..29065b1 100644 --- a/REUSE.md +++ b/REUSE.md @@ -147,6 +147,7 @@ | `agentapi.StatusError` | controller/internal/agentapi/client.go | `{Path, Code}` typed non-2xx GET error | Distinguishing HTTP statuses from transport errors (`errors.As`) | NEVER string-match agent error text — the capability probe keys on `Code==404` | | `SupportCache.Supports` / `Client.Supports` | controller/internal/agentapi/features.go | `(ctx, prober, Feature) SupportState` | Agent-capability gate for COUPLED features (route probe, TTL 5m) | 404 ⇒ No; transport/5xx ⇒ Unknown (NEVER refuse on Unknown). New coupled feature = new `featureProbes` row + gate call at the entry point + `MinAgent:` in the CHANGELOG header (publish-train-rules.md). Web layer: `Server.netFeatures` through the `netAgent` seam | | `agentapi.DiskVerdictFor` / `DiskVerdict.Label` / `DegradedAttributes` | controller/internal/agentapi/diskverdict.go | `(*SmartSummary) DiskVerdict` | THE shared disk-health verdict (card chip + 6h check) — v0.169.0 | Pure; nil/UNKNOWN → `DiskVerdictUnknown` (Nincs adat, NEVER alarms); percentage_used threshold is **≥90**. Feature-detects the agent's `DiskInfo.Smart` (nil = old agent). Do NOT recompute the verdict inline anywhere | +| `Server.resolveBackupTargetState` / `backupTargetView` | controller/internal/web/backup_target_offer.go | `(ctx)` → state / `*BackupTargetView` (nil = render nothing) | The whole-system backup-target answer: healthy · degraded-never-configured · **TargetAbsent** (configured, drive gone) · unknown | Test seams `Server.tiersFn` + `Server.disksFn` (nil → the real client). **`degradedMessageFor` is the ONE place that decides customer copy** — add a state there, never in a template. `backupTargetView` returns **nil** for healthy AND unknown so a template typo cannot decorate a working box. R-112: this state had NO consumer for two releases; the render is server-side on `backups.html`, and the seam test drives `backupsHandler` and asserts rendered HTML | | `Server.cachedDisks` / `RunDiskHealthCheck` | controller/internal/web/disk_health.go | `(ctx)` | Card fetch (60s TTL) / the 6h degradation check | Card uses the 60s TTL cache (anti-smartctl-storm); the CHECK fetches FRESH (`fetchDisks`). Test seams: `Server.disksFn` (source) + `Server.diskNotifyFn` (sink). Baseline is in-memory (restart re-baselines) | ### Notifications / hub sync diff --git a/controller/README.md b/controller/README.md index 0bdd440..ca03bae 100644 --- a/controller/README.md +++ b/controller/README.md @@ -633,12 +633,28 @@ height with no magic number to drift as item counts change. | Route | Page | Sections | |-------|------|----------| -| `/backups` | Áttekintés | storage overview, whole-guest Rendszermentés, status stat cards, single-copy warning | +| `/backups` | Áttekintés | storage overview, whole-guest Rendszermentés, status stat cards, single-copy warning, **backup-target banner + offer (v0.186.0)** | | `/backups/remote` | Távoli mentés | Felhom-offsite status card (3 states, display-only), tier-3 status block + quota, participation toggles (+ zero-toggle hint; the persisted zero-toggle run-warning is DISPLAY-replaced by a "kijelölés módosult" note once ≥1 app is toggled — `offboxWarningDisplay`, v0.126.0), manual-target form (`#offbox-section`) | | `/backups/apps` | Alkalmazások | schedule, Adatbázisok table, per-app 1./2./3. tier rows (tier-2 config entry; tier-3 actions deep-link to `/backups/remote#offbox-section`) | | `/backups/restore` | Visszaállítás | restore panel, offsite restore list (**one „Visszaállítás…" entry per app** since v0.154.0), existing verification copies, .fab download/import loop | | `/backups/restore/app?name=` | Visszaállítás — | **R-48 per-app offsite restore wizard** (v0.154.0). GET-only; three described intent cards (ellenőrzés / hiányzó fájlok / teljes visszaállítás), a visible phase strip, and a server-derived step. Adds NO mutation endpoint — every card posts to the pre-existing `/backup/offbox/{restore,place,reconstitute}` | +**Backup-target banner + offer (E-2 · v0.186.0, R-114 + R-112).** The `/backups` page renders the +whole-system backup-target state server-side, from the AGENT's view (never from our own intent flag). +Four outcomes, three of which the customer sees nothing for or one thing for: + +| State | Renders | +|---|---| +| healthy — a real drive holds the target | **nothing** (no badge, no reassurance: a working box must look normal) | +| degraded, never configured (`local`/unset) | the system-disk copy **+ an offer control** that POSTs `/api/storage/backup-target/assign` | +| **configured, drive absent** (`TargetAbsent`) | the absent-drive copy, **no offer** — the remedy is to reconnect *that* drive | +| unknown (agent unreachable / pre-R-82) | **nothing** — absence of an answer is not degradation | + +`degradedMessageFor` is the single decision point for customer copy; `backupTargetView` returns nil +for the two silent states. The absent copy is verbatim the hub's `backup_target_absent` email so the +banner and the mail agree. The offer never auto-submits, and `restart_required` from assign is shown +rather than papered over with a self-restart (the agent deliberately does not restart itself). + Shared data builders: `backupsCommonData` (chrome + full-status + flash) + `backupsOffboxData` (offbox target/toggles) in `handlers.go`; shared partials in `templates/backups_shared.html`. (The v0.124.0 split was MOVE-only, gated one-shot by `backups_split_move_check.py`; the gate was diff --git a/controller/internal/web/backup_target_absent_test.go b/controller/internal/web/backup_target_absent_test.go new file mode 100644 index 0000000..9a0f64f --- /dev/null +++ b/controller/internal/web/backup_target_absent_test.go @@ -0,0 +1,193 @@ +package web + +import ( + "context" + "strings" + "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/agentapi" + "gitea.dooplex.hu/admin/felhom-controller/internal/settings" +) + +// R-114 — the third state: CONFIGURED, and its drive is GONE. +// +// THE BUG THESE PIN, measured live in E-2d (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.3). With +// the assigned target's drive detached, the endpoint returned: +// +// {"degraded":true,"target":"felhom-backup", +// "message":"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer …", +// "offer_path":"/mnt/felhom-drives/mentes2","offer_label":"Mentés meghajtó"} +// +// Two falsehoods in one payload. The backup was NOT on the system disk — it was on a drive that had +// vanished. And the remedy offered was THE DRIVE THAT JUST DISAPPEARED. resolveBackupTargetState had +// only two outcomes (a disk claims the target, or nothing does) so the third state fell into the +// second and inherited its message and its offer. +// +// The fixtures below are the observed shapes, not invented ones: the drive is still listed with a +// MountPath (the registry keeps the configured path), Role user-data, but BackupTarget is false — +// which is exactly why the healthy branch missed it and the offer branch matched it. + +// absentHarness wires a Server whose agent reports `primary` as the tier target and `disks` as the +// drive list, with one registered storage path so firstOfferableDrive has something to find. Both +// halves come through the production seams (tiersFn / disksFn), so the resolver under test is the +// real one. +func absentHarness(t *testing.T, primary string, disks []agentapi.DiskInfo, registerPath string) *Server { + t.Helper() + s := testServer(t) + if registerPath != "" { + if err := s.settings.AddStoragePath(settings.StoragePath{Path: registerPath, Label: "Mentés meghajtó"}); err != nil { + t.Fatalf("register storage path: %v", err) + } + } + s.tiersFn = func(context.Context) (agentapi.TiersResponse, error) { + return agentapi.TiersResponse{Tiers: []agentapi.BackupTierInfo{{Target: primary, Primary: true}}}, nil + } + s.disksFn = func(context.Context) (agentapi.DisksResponse, error) { + return agentapi.DisksResponse{Disks: disks}, nil + } + return s +} + +// theVanishedDrive is the E-2d shape: registered, still carrying its configured MountPath, user-data, +// but no longer claiming the backup target because its device is gone. +var theVanishedDrive = agentapi.DiskInfo{ + Name: "mentes2", + MountPath: "/mnt/mentes2", + GuestPath: "/mnt/felhom-drives/mentes2", + Role: "user-data", + BackupTarget: false, +} + +// ── Scenario C — configured, drive absent ─────────────────────────────────────────────────────── + +// RED-PROOF: delete the `if targetIsConfiguredDrive(primary)` branch from resolveBackupTargetState +// and this fails twice — the message becomes the system-disk copy and OfferPath becomes non-empty. +func TestConfiguredButAbsentTargetDoesNotClaimTheSystemDisk(t *testing.T) { + s := absentHarness(t, "felhom-backup", []agentapi.DiskInfo{theVanishedDrive}, "/mnt/felhom-drives/mentes2") + st := s.resolveBackupTargetState(context.Background()) + + if !st.Known { + t.Fatal("state is unknown — the fixture answers both agent calls") + } + if !st.TargetAbsent { + t.Fatalf("TargetAbsent is false for target %q with no disk claiming it — the configured-but-gone "+ + "state fell back into never-configured, which is the R-114 bug", st.TargetID) + } + msg := degradedMessageFor(st) + if strings.Contains(msg, "ugyanazon a lemezen") { + t.Errorf("the customer is told the backup is on the SYSTEM DISK, which is false — the target is "+ + "%q, a drive that has vanished. Got: %s", st.TargetID, msg) + } + if !strings.Contains(msg, "nem érhető el") { + t.Errorf("the absent-drive copy is missing; got: %s", msg) + } +} + +// The remedy must be "reconnect THAT drive", never "pick a different one" — and above all never the +// drive that just disappeared, which is what E-2d actually observed being offered. +func TestConfiguredButAbsentTargetOffersNothing(t *testing.T) { + s := absentHarness(t, "felhom-backup", []agentapi.DiskInfo{theVanishedDrive}, "/mnt/felhom-drives/mentes2") + st := s.resolveBackupTargetState(context.Background()) + + if st.OfferPath != "" { + t.Errorf("offered %q while the configured target is absent — E-2d observed this exact payload "+ + "offering the drive that had just vanished", st.OfferPath) + } + if st.OfferLabel != "" { + t.Errorf("offer label %q leaked with no offer path", st.OfferLabel) + } +} + +// The suppression must be the branch's own doing, NOT a side effect of the drive being flagged +// Disconnected — that flag is set by the agent-side drive gate in another repo (R-113), and this +// state has to be correct before, during and independently of that landing. +func TestAbsentTargetSuppressesOfferWithoutRelyingOnTheDisconnectedFlag(t *testing.T) { + s := absentHarness(t, "felhom-backup", []agentapi.DiskInfo{theVanishedDrive}, "/mnt/felhom-drives/mentes2") + for _, sp := range s.settings.GetStoragePaths() { + if sp.Disconnected { + t.Fatalf("fixture invalid: %s is already marked Disconnected, so this test would pass "+ + "for the wrong reason", sp.Path) + } + } + if st := s.resolveBackupTargetState(context.Background()); st.OfferPath != "" { + t.Errorf("offer %q survived with Disconnected UNSET — the suppression is leaning on the "+ + "other repo's flag instead of on this state", st.OfferPath) + } +} + +// ── Scenario A — never configured (must be unchanged) ─────────────────────────────────────────── + +func TestNeverConfiguredStillSaysSystemDiskAndStillOffers(t *testing.T) { + s := absentHarness(t, builtinLocalTarget, []agentapi.DiskInfo{{ + Name: "hdd1", MountPath: "/mnt/hdd1", GuestPath: "/mnt/felhom-drives/hdd1", Role: "user-data", + }}, "/mnt/felhom-drives/hdd1") + st := s.resolveBackupTargetState(context.Background()) + + if st.TargetAbsent { + t.Fatal("target `local` marked absent — the builtin root-fs storage is the NEVER-CONFIGURED " + + "state, not a drive that went missing") + } + if !st.Degraded { + t.Fatal("a backup on the system disk must still read degraded") + } + if msg := degradedMessageFor(st); !strings.Contains(msg, "ugyanazon a lemezen") { + t.Errorf("the never-configured copy changed; got: %s", msg) + } + if st.OfferPath == "" { + t.Error("no offer in the never-configured state — this is exactly where an offer belongs") + } +} + +// An unset tier is the same customer situation as `local`: nothing chosen yet, so it must not be +// reported as a drive that vanished. +func TestUnsetTierIsNeverConfiguredNotAbsent(t *testing.T) { + if targetIsConfiguredDrive("") { + t.Error("an empty target id was treated as a configured drive") + } + if targetIsConfiguredDrive(builtinLocalTarget) { + t.Error("the builtin `local` was treated as a configured drive") + } + if !targetIsConfiguredDrive("felhom-backup") { + t.Error("a real storage id was not treated as a configured drive") + } +} + +// ── Scenario B — healthy still renders nothing ────────────────────────────────────────────────── + +func TestHealthyTargetIsNeitherDegradedNorAbsent(t *testing.T) { + s := absentHarness(t, "felhom-backup", []agentapi.DiskInfo{{ + Name: "mentes2", MountPath: "/mnt/mentes2", GuestPath: "/mnt/felhom-drives/mentes2", + Role: "user-data", BackupTarget: true, + }}, "/mnt/felhom-drives/mentes2") + st := s.resolveBackupTargetState(context.Background()) + + if st.Degraded || st.TargetAbsent { + t.Fatalf("a healthy target reported degraded=%v absent=%v", st.Degraded, st.TargetAbsent) + } + if msg := degradedMessageFor(st); msg != "" { + t.Errorf("healthy produced copy %q — a working box must look normal", msg) + } +} + +// ── the copy contract ─────────────────────────────────────────────────────────────────────────── + +// The absent copy is VERBATIM the hub's customerMessages["backup_target_absent"]. If either side is +// reworded this fails, which is the only thing currently binding the two repos' strings together. +func TestAbsentCopyMatchesTheHubEmailWordForWord(t *testing.T) { + const hubCopy = "A rendszermentés meghajtója nem érhető el — amíg vissza nem " + + "csatlakoztatod, a teljes rendszermentés nem készül el." + if backupTargetAbsentText != hubCopy { + t.Errorf("the banner and the email now tell the customer different stories about one drive.\n"+ + " banner: %s\n email : %s", backupTargetAbsentText, hubCopy) + } + msg := degradedMessageFor(BackupTargetState{Known: true, Degraded: true, TargetAbsent: true}) + for _, want := range []struct{ frag, why string }{ + {"nem érhető el", "the FACT — the drive cannot be reached"}, + {"nem készül el", "the CONSEQUENCE — the full system backup does not happen"}, + {"vissza nem csatlakoztatod", "the REMEDY — reconnect it"}, + } { + if !strings.Contains(msg, want.frag) { + t.Errorf("absent copy is missing %s (%q); got: %s", want.why, want.frag, msg) + } + } +} diff --git a/controller/internal/web/backup_target_offer.go b/controller/internal/web/backup_target_offer.go index ee9c239..e1a8605 100644 --- a/controller/internal/web/backup_target_offer.go +++ b/controller/internal/web/backup_target_offer.go @@ -21,9 +21,22 @@ type BackupTargetState struct { // Known is false when the agent could not be asked. Everything below is then meaningless, and the // UI must render NOTHING rather than guess — an unreachable agent is not evidence of degradation. Known bool - // Degraded is true when the whole-guest backup lands on the SYSTEM drive, so it protects against - // corruption but not against drive loss. + // Degraded is true when the whole-guest backup is NOT protected against drive loss. It answers + // "is there a problem", not "which problem" — TargetAbsent below is the discriminator. Both + // problem states set it, so the wire's `degraded` flag keeps its meaning for every consumer. Degraded bool + // TargetAbsent (R-114) separates the two problem states, which have OPPOSITE remedies: + // + // Degraded && !TargetAbsent — never configured. The backup is on the system drive. Remedy: + // attach a second drive and assign it. An offer belongs here. + // Degraded && TargetAbsent — configured, and its drive is GONE. The backup is not on the + // system drive at all. Remedy: reconnect THAT drive. No offer — + // suggesting a different drive is the wrong instruction. + // + // Before R-114 the second case fell into the first, so a customer whose backup drive had vanished + // was told the backup was on the system disk (false) and offered the drive that just disappeared + // (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.3). + TargetAbsent bool // TargetID is the agent's primary tier storage id (e.g. "felhom-backup" / "local"). TargetID string // Label is the customer-facing drive name when the target is a real drive. @@ -43,11 +56,7 @@ type BackupTargetState struct { // a different question. This state answers only "is there a LOCAL copy that survives a disk failure", // which is matrix row 4. func (s *Server) resolveBackupTargetState(ctx context.Context) BackupTargetState { - agent, err := s.agentClient() - if err != nil { - return BackupTargetState{} - } - tiers, err := agent.BackupTiers(ctx) + tiers, err := s.fetchBackupTiers(ctx) if err != nil { // Includes ErrTiersUnsupported (a pre-R-82 agent). Unknown, never "degraded": claiming // degradation because we could not ask would put a permanent warning on a healthy box. @@ -62,7 +71,7 @@ func (s *Server) resolveBackupTargetState(ctx context.Context) BackupTargetState } st := BackupTargetState{Known: true, TargetID: primary} - disks, derr := agent.Disks(ctx) + disks, derr := s.fetchDisks(ctx) if derr != nil { return BackupTargetState{} // could not classify → say nothing } @@ -73,13 +82,48 @@ func (s *Server) resolveBackupTargetState(ctx context.Context) BackupTargetState return st } } - // No disk claims the target ⇒ it is the builtin `local` on the system drive, or the tier is - // unset. Either way the local backup does not survive drive loss. + // No disk claims the target. Two DIFFERENT states land here and R-114 separates them, because + // before it they shared one message and one remedy — and for the second the message was false. st.Degraded = true + if targetIsConfiguredDrive(primary) { + // A real storage id is configured, yet no disk claims it ⇒ its drive is GONE. Saying "the + // backup is on the system disk" here is simply untrue, and offering another drive answers a + // question the customer did not ask. The remedy is to reconnect THAT drive. + st.TargetAbsent = true + // Offer suppressed unconditionally — NOT left to firstOfferableDrive's Disconnected skip. + // That skip only works once the agent's drive-gate has marked the path (R-113, another repo); + // this state must be correct on its own. Belt here, braces there. + return st + } st.OfferPath, st.OfferLabel = s.firstOfferableDrive(disks.Disks) return st } +// fetchBackupTiers reads the agent's tier view through the tiersFn test seam (nil → the real client), +// mirroring fetchDisks so both halves of this state come from seams a test can drive. +func (s *Server) fetchBackupTiers(ctx context.Context) (agentapi.TiersResponse, error) { + if s.tiersFn != nil { + return s.tiersFn(ctx) + } + client, err := s.agentClient() + if err != nil { + return agentapi.TiersResponse{}, err + } + return client.BackupTiers(ctx) +} + +// builtinLocalTarget is the PVE builtin storage on the root filesystem — the "no separate drive" +// target. Anything else is a deliberately configured storage. +const builtinLocalTarget = "local" + +// targetIsConfiguredDrive reports whether the primary tier names a real storage rather than the +// builtin root-fs one. Empty means the tier is unset (never configured), which is the same customer +// situation as `local`: nothing has been chosen yet. +func targetIsConfiguredDrive(targetID string) bool { + t := strings.TrimSpace(targetID) + return t != "" && t != builtinLocalTarget +} + // firstOfferableDrive picks a registered, connected, non-network drive that could hold the backup. // It is a SUGGESTION for the offer — the customer still has to choose (E-2 §3). Nothing here assigns // anything. @@ -120,6 +164,15 @@ const ( "így hibás fájlok ellen véd, lemezhiba ellen nem. Csatlakoztass egy második meghajtót a teljes védelemhez." backupTargetOfferText = "Ezt a meghajtót kijelölheted a rendszermentés helyéül — így egy lemezhiba " + "után is vissza tudod állítani a rendszert." + // backupTargetAbsentText (R-114) is the CONFIGURED-BUT-GONE state. It is VERBATIM the hub's + // customerMessages["backup_target_absent"] (felhom.eu hub/internal/notify/templates.go:93) so the + // banner a customer reads on the page and the email they receive say exactly the same thing — a + // customer who is told two different stories about one drive trusts neither. + // + // DRIFT RISK, filed not fixed: this string now lives in two repos with nothing binding them. If + // one is reworded the other silently disagrees. + backupTargetAbsentText = "A rendszermentés meghajtója nem érhető el — amíg vissza nem " + + "csatlakoztatod, a teljes rendszermentés nem készül el." ) // degradedMessageFor is the single decision point for "does the customer see anything?" — extracted @@ -133,11 +186,56 @@ func degradedMessageFor(st BackupTargetState) string { if !st.Known || !st.Degraded { return "" } + if st.TargetAbsent { + // R-114: configured, drive gone. A different fact with a different remedy, so a different + // sentence — routed through here so there is still exactly one place that decides copy. + return backupTargetAbsentText + } return backupTargetDegradedText } -// handleBackupTargetState serves GET /api/storage/backup-target — the dashboard's source for the degraded -// banner and the offer. +// ---- the render (R-112) ------------------------------------------------------------------------ + +// BackupTargetView is the template-facing shape of this state. It exists so the template stays a +// dumb renderer: every "does the customer see anything?" decision is already made by the time it +// arrives, in degradedMessageFor, which remains the single decision point. +// +// R-112: until v0.186.0 this state had NO consumer at all. The endpoint was byte-correct and no +// template, handler or script ever asked for it — the controller's templates fetch 18 distinct +// /api/storage/* endpoints and backup-target was one of the only two with zero references, so a +// customer whose backup was unprotected was never told (felhom.eu +// audits/E2D-fresh-vm-2026-07-29.md §5.1). Server-rendered here rather than a 19th fetch: the state +// is already resolved when the page is built, and a banner that needs JavaScript to appear is one +// more thing that can silently not happen. +type BackupTargetView struct { + // Message is the customer copy. Empty is impossible here — a nil *BackupTargetView means + // "render nothing", so the template never has to decide. + Message string + // OfferPath is empty in every state except never-configured-with-an-eligible-drive. + OfferPath string + OfferLabel string + OfferText string +} + +// backupTargetView resolves the state and reduces it to what the page renders, or nil for the two +// states that render NOTHING — healthy and unknown. Returning nil rather than an empty struct means +// a template typo cannot accidentally decorate a working box. +func (s *Server) backupTargetView(ctx context.Context) *BackupTargetView { + st := s.resolveBackupTargetState(ctx) + msg := degradedMessageFor(st) + if msg == "" { + return nil // healthy or unknown — a working configuration must look normal + } + v := &BackupTargetView{Message: msg} + if st.OfferPath != "" { + v.OfferPath, v.OfferLabel, v.OfferText = st.OfferPath, st.OfferLabel, backupTargetOfferText + } + return v +} + +// handleBackupTargetState serves GET /api/storage/backup-target — the JSON view of the same state the +// backups page renders server-side (see backupTargetView). Kept because `assign` needs a POST partner +// and the payload is a stable contract. func (s *Server) handleBackupTargetState(w http.ResponseWriter, r *http.Request) { st := s.resolveBackupTargetState(r.Context()) out := map[string]any{"known": st.Known} diff --git a/controller/internal/web/backup_target_render_test.go b/controller/internal/web/backup_target_render_test.go new file mode 100644 index 0000000..96f8e42 --- /dev/null +++ b/controller/internal/web/backup_target_render_test.go @@ -0,0 +1,145 @@ +package web + +import ( + "context" + "net/http/httptest" + "strings" + "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/agentapi" + "gitea.dooplex.hu/admin/felhom-controller/internal/settings" +) + +// R-112 — SCENARIO E: THE SEAM. This is the test whose absence let E-2 ship. +// +// The endpoint was byte-correct the whole time. `resolveBackupTargetState` produced the right state, +// `degradedMessageFor` produced the right Hungarian, and v0.185.1 even added a test proving the route +// dispatched. None of that was ever reachable by a customer, because NOTHING fetched or rendered it: +// the controller's templates fetch 18 distinct /api/storage/* endpoints and backup-target was one of +// only two with zero references (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.1). +// +// So these tests deliberately do NOT call the resolver and assert its string — that proves the +// resolver, which was never broken. They drive `backupsHandler` over httptest and assert the RENDERED +// HTML, travelling handler → backupTargetView → resolveBackupTargetState → degradedMessageFor → +// the production "backups" template. Delete the one line in backupsHandler that sets +// data["BackupTarget"] and every assertion below fails, which is precisely the class of regression +// that shipped five times in this project. + +// renderBackupsPage drives the REAL page handler and returns the HTML a browser would receive. +func renderBackupsPage(t *testing.T, primary string, disks []agentapi.DiskInfo, registerPath string) string { + t.Helper() + s := absentHarness(t, primary, disks, registerPath) + s.loadTemplates() + rr := httptest.NewRecorder() + s.backupsHandler(rr, httptest.NewRequest("GET", "/backups", nil)) + if rr.Code != 200 { + t.Fatalf("backups page returned %d, want 200", rr.Code) + } + return rr.Body.String() +} + +// C — configured, drive absent: the truthful copy reaches the page, and no offer does. +func TestBackupsPageRendersTheAbsentDriveCopy(t *testing.T) { + html := renderBackupsPage(t, "felhom-backup", []agentapi.DiskInfo{theVanishedDrive}, "/mnt/felhom-drives/mentes2") + + if !strings.Contains(html, backupTargetAbsentText) { + t.Error("the absent-drive copy never reached the page — the customer is told nothing while " + + "their backup drive is missing (R-112: the state had no consumer at all)") + } + if strings.Contains(html, backupTargetDegradedText) { + t.Error("the page shows the SYSTEM-DISK copy for an absent configured drive (R-114)") + } + // Assert the CONTROL's markup, not the bare id: the page script always contains + // getElementById('backup-target-assign'), so a substring match on the id alone matches the + // script and reports a control that is not there. (This test caught exactly that on itself.) + if strings.Contains(html, `id="backup-target-assign"`) { + t.Error("an offer control rendered while the configured target is absent — the remedy is to " + + "reconnect that drive, not to pick another") + } + if strings.Contains(html, `id="backup-target-offer"`) { + t.Error("the offer block rendered while the configured target is absent") + } +} + +// A — never configured: the copy AND the offer control both render, and the control carries the +// drive's path so accepting it can only ever assign that one. +func TestBackupsPageRendersTheOfferWhenNeverConfigured(t *testing.T) { + html := renderBackupsPage(t, builtinLocalTarget, []agentapi.DiskInfo{{ + Name: "hdd1", MountPath: "/mnt/hdd1", GuestPath: "/mnt/felhom-drives/hdd1", Role: "user-data", + }}, "/mnt/felhom-drives/hdd1") + + if !strings.Contains(html, backupTargetDegradedText) { + t.Error("the degraded copy never reached the page") + } + if !strings.Contains(html, backupTargetOfferText) { + t.Error("the offer copy never reached the page") + } + if !strings.Contains(html, `id="backup-target-assign"`) { + t.Error("no offer control rendered — the customer is told to attach a drive but given no way " + + "to assign the one they already have") + } + if !strings.Contains(html, `data-path="/mnt/felhom-drives/hdd1"`) { + t.Error("the offer control does not carry the offered drive's path") + } + // It is an OFFER: it must not submit itself. + if strings.Contains(html, "backup-target-assign.click()") || strings.Contains(html, "autosubmit") { + t.Error("the offer control auto-submits — declining must be possible by doing nothing") + } +} + +// B — healthy renders NOTHING. Assert the absence of all three strings, not the absence of an error. +func TestBackupsPageRendersNothingWhenHealthy(t *testing.T) { + html := renderBackupsPage(t, "felhom-backup", []agentapi.DiskInfo{{ + Name: "mentes2", MountPath: "/mnt/mentes2", GuestPath: "/mnt/felhom-drives/mentes2", + Role: "user-data", BackupTarget: true, + }}, "/mnt/felhom-drives/mentes2") + + assertNoBackupTargetCopy(t, html, "a HEALTHY box grew a banner — a working configuration must "+ + "look normal, or every dashboard carries a permanent notice and warnings stop being read") +} + +// D — unknown renders NOTHING. Not being able to ask the agent is not evidence of degradation. +func TestBackupsPageRendersNothingWhenAgentUnreachable(t *testing.T) { + s := testServer(t) + if err := s.settings.AddStoragePath(settings.StoragePath{Path: "/mnt/felhom-drives/hdd1"}); err != nil { + t.Fatalf("register: %v", err) + } + // No tiersFn/disksFn seams and no configured endpoint → agentClient() fails → Known:false. + s.loadTemplates() + rr := httptest.NewRecorder() + s.backupsHandler(rr, httptest.NewRequest("GET", "/backups", nil)) + + assertNoBackupTargetCopy(t, rr.Body.String(), "an UNREACHABLE agent produced a customer warning — "+ + "absence of an answer is not evidence of degradation (R-88 Part 2's mistake)") +} + +func assertNoBackupTargetCopy(t *testing.T, html, why string) { + t.Helper() + for _, s := range []struct{ frag, name string }{ + {backupTargetDegradedText, "the system-disk copy"}, + {backupTargetAbsentText, "the absent-drive copy"}, + {backupTargetOfferText, "the offer copy"}, + } { + if strings.Contains(html, s.frag) { + t.Errorf("%s rendered: %s", s.name, why) + } + } + if strings.Contains(html, `id="backup-target-alert"`) { + t.Errorf("the banner element rendered: %s", why) + } +} + +// The view is nil — not an empty struct — in the two silent states, so a template typo cannot +// accidentally decorate a working box with an empty alert box. +func TestBackupTargetViewIsNilWhenNothingShouldRender(t *testing.T) { + healthy := absentHarness(t, "felhom-backup", []agentapi.DiskInfo{{ + MountPath: "/mnt/mentes2", GuestPath: "/mnt/felhom-drives/mentes2", Role: "user-data", BackupTarget: true, + }}, "") + if v := healthy.backupTargetView(context.Background()); v != nil { + t.Errorf("healthy returned a non-nil view %+v — nil is what makes the template render nothing", v) + } + unknown := testServer(t) + if v := unknown.backupTargetView(context.Background()); v != nil { + t.Errorf("unknown returned a non-nil view %+v", v) + } +} diff --git a/controller/internal/web/handlers.go b/controller/internal/web/handlers.go index c6cf9bf..17ac3c9 100644 --- a/controller/internal/web/handlers.go +++ b/controller/internal/web/handlers.go @@ -840,6 +840,11 @@ func (s *Server) backupsHandler(w http.ResponseWriter, r *http.Request) { // Whole-guest backup view (agent-sourced, read-only) for the "Rendszermentés" section. data["GuestBackup"] = s.loadGuestBackup(r.Context()) + // R-112: the backup-target banner finally has a consumer. nil in the healthy and unknown states, + // so the template renders nothing at all — no badge, no reassurance. This is the seam whose + // absence made E-2's degraded banner and offer invisible to every customer. + data["BackupTarget"] = s.backupTargetView(r.Context()) + // Customer-configurable backup window (v0.168.0): effective start + derived leg/gate times. s.backupWindowData(data) diff --git a/controller/internal/web/server.go b/controller/internal/web/server.go index 4477821..da845a2 100644 --- a/controller/internal/web/server.go +++ b/controller/internal/web/server.go @@ -156,6 +156,13 @@ type Server struct { disksFn func(context.Context) (agentapi.DisksResponse, error) diskNotifyFn func(label string, attrs []string, critical bool) + // tiersFn is the sibling test seam for the agent's backup-tier view (nil → the real client's + // BackupTiers()). Added with R-114 so the backup-target state — which is the source of a + // customer-facing banner — is testable through its REAL resolver rather than only through the + // pure copy helper. Without it the resolver's own branching had no test at all, which is how the + // configured-but-absent case reached production saying the wrong thing. + tiersFn func(context.Context) (agentapi.TiersResponse, error) + // App-email SMTP shim lifecycle (optional — nil when no hub is configured or the kill-switch is // off). The global app-email settings toggle calls Apply() so the shim starts/stops at runtime. mailShim MailShimController diff --git a/controller/internal/web/templates/backups.html b/controller/internal/web/templates/backups.html index 800e888..252be46 100644 --- a/controller/internal/web/templates/backups.html +++ b/controller/internal/web/templates/backups.html @@ -11,6 +11,21 @@
{{.Backup.SingleCopyWarning}}
{{end}}{{end}} +{{/* R-112: the whole-system backup-target state. nil = healthy or unknown = render NOTHING. */}} +{{if .BackupTarget}} +
{{.BackupTarget.Message}}
+{{if .BackupTarget.OfferPath}} +
+

{{.BackupTarget.OfferText}}

+

{{.BackupTarget.OfferLabel}}

+ +
+
+{{end}} +{{end}} + {{if not .Backup}} {{template "backups_empty" .}} {{else}} @@ -215,6 +230,43 @@ function pollGuestBackup(out, btn) { if (tries > 180) { clearInterval(iv); btn.disabled = false; } // ~15 min cap at 5s polls }, 5000); } + +// R-112 — ACCEPTING the offer. It is an OFFER: this runs only on an explicit click behind the +// standard inline confirm, and declining is simply never calling it. Nothing else on this page may +// write the role, and there is no auto-submit. +(function () { + var btn = document.getElementById('backup-target-assign'); + if (!btn) { return; } // no offer in this state — nothing to wire + btn.addEventListener('click', function () { + var out = document.getElementById('backup-target-assign-result'); + btn.disabled = true; + out.innerHTML = '

Kijelölés folyamatban…

'; + fetch('/api/storage/backup-target/assign', { + method: 'POST', + headers: Object.assign({ 'Content-Type': 'application/json' }, csrfHeaders()), + body: JSON.stringify({ path: btn.getAttribute('data-path') }) + }) + .then(function (r) { return r.json(); }) + .then(function (j) { + if (!j.ok) { + out.innerHTML = '
A kijelölés nem sikerült: ' + (j.error || '') + '
'; + btn.disabled = false; + return; + } + // The agent deliberately does NOT restart itself — restarting with a backup in flight + // records a spurious tier failure. So say so plainly instead of hiding it or faking it. + if (j.data && j.data.restart_required) { + out.innerHTML = '
A meghajtó kijelölve. A beállítás a host-ügynök következő újraindulása után lép életbe.
'; + } else { + out.innerHTML = '
A meghajtó kijelölve.
'; + } + }) + .catch(function (e) { + out.innerHTML = '
Hiba: ' + e.message + '
'; + btn.disabled = false; + }); + }); +})(); {{template "layout_end" .}}