From 811a0ef7a7f1b9249c3551536a6385479f678baf Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 13 Jul 2026 10:17:58 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20v0.124.0=20backups=20IA=20restructure?= =?UTF-8?q?=20=E2=80=94=20CHANGELOG=20+=20REPORT=20+=20CONTEXT=20(decision?= =?UTF-8?q?s=201-3)=20+=20README=20page=20map=20+=20REUSE;=20records=20the?= =?UTF-8?q?=20HIGH=20containerized-.fab=20volume-strand=20finding=20and=20?= =?UTF-8?q?the=20agent=20SystemDisks=20legacy-boot=20finding?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_01GzammAMzsJTgpQHqxwM2bC --- CHANGELOG.md | 40 +++++++++++++ CONTEXT.md | 17 +++++- REPORT.md | 130 +++++++++++++++++++++++++------------------ REUSE.md | 2 + controller/README.md | 19 ++++++- 5 files changed, 151 insertions(+), 57 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a3c7030..ee38106 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,45 @@ ## Changelog +### v0.124.0 — backups IA restructure: four sub-pages, Felhom-offsite status card, .fab browser download (2026-07-13) — MinAgent: 0.81.0 + +The nine-section backups page split into four sub-pages (operator review: customers got lost); +plus the offsite status card and the .fab portability exit. Controller-only; floor untouched. +Operator decisions recorded in CONTEXT: single active offsite destination stands; the status +card never changes state; .fab is portability, NOT a backup tier. + +- **IA split** (`backups{,_remote,_apps,_restore}.html` + `backups_shared.html` partials): + **Áttekintés** `/backups` (storage overview, Rendszermentés, stat cards, single-copy warning), + **Távoli mentés** `/backups/remote` (status card + toggles + manual-target form, + `#offbox-section` anchor), **Alkalmazások** `/backups/apps` (schedule, Adatbázisok, per-app + 1./2./3. rows), **Visszaállítás** `/backups/restore` (restore panel, the RELOCATED offbox + restore-to-verify, the .fab loop). Sections MOVED verbatim — `scripts/backups_split_move_check.py` + compares all 15 blocks against the v0.123.0 baseline (red-proven). Shared data builder + extracted (`backupsCommonData` + `backupsOffboxData`) — no duplicated computation. Old links + survive: `/backups` = Áttekintés; tier-3 row actions → `/backups/remote#offbox-section`; + offbox/restore/tier2 flash redirects + the tier2-config back-link retargeted per page. +- **Felhom-offsite status card** (top of Távoli mentés; local data only, DISPLAY-ONLY — no form, + no button, unit-enforced): (1) no target → "Felhom offsite tárhely — igényelhető szolgáltatás. + …Érdeklődj az üzemeltetőnél."; (2) applied + zero toggled → "Aktív — nincs kijelölt + alkalmazás" (+ the v0.123.0 hint below); (3) applied + toggled → no card (the status block is + the state). States 2 and 3 live-proven on the drill box. +- **.fab browser download** (`handler_export_download.go`): the EXISTING async export pipeline + with dest = a staging dir under the data dir (`fab-downloads/`; same producer → byte-identical + bundle), estimate shown BEFORE start, then a guarded streaming exit + (`GET /api/export/download?file=` — basename-shape + dir-containment guard, red-proven against + prefix-only matching; `io.Copy`, `Content-Disposition: attachment`, post-stream removal, 1h TTL + sweep on startup + each start). Batch UI downloads apps ONE AT A TIME (no mega-zip). Import + stays drive-scan; portability copy on the section ("Hordozható pillanatfelvétel…"). + Unit round-trip: real export → import → content equality; a corrupted bundle is REFUSED + (gzip CRC — there is no per-file checksum; documented). +- **FINDINGS from the live §13 run** (both pre-existing, recorded for follow-up tasks): + **(HIGH)** containerized-controller .fab export of Docker-VOLUME apps strands the volume tar on + the GUEST host (`docker run -v :/out` resolves against the host FS) → the bundle + ships an EMPTY `data/volumes`, export reports success, import brings the app up EMPTY. HDD-data + apps unaffected. Live-hit on demo (ActualBudget; data restored from the stranded tar by hand). + **(MEDIUM)** agent-side: on a legacy-boot PVE with LVM root, `SystemDisks` resolves NO raw + system disk → `sysKnown=false` → every disk classified system → the drive wizard can never + offer a candidate (drill box; hot-added disk invisible). + ### v0.123.0 — polish batch: F-15 instant reset codes, F-11 inline confirms, Tier-3 "Távoli mentés" rename, zero-toggle honesty (2026-07-13) — MinAgent: 0.81.0 Four independent fixes from the take-two drill + operator review. Live-validated on the drill box diff --git a/CONTEXT.md b/CONTEXT.md index 4a5e435..8ba5b62 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,22 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-13 (v0.123.0 — polish batch) +Last updated: 2026-07-13 (v0.124.0 — backups IA restructure) + +> **2026-07-13 — v0.124.0: BACKUPS IA RESTRUCTURE. MinAgent 0.81.0 + floor unchanged. +> Operator decisions (2026-07-13, treat as settled):** (1) single active offsite destination per +> box STANDS — the dual-destination `managed_by` model is the separate queued Task B; +> (2) the Felhom-offsite status card NEVER changes anything — display + opt-in pointers only; +> (3) .fab export/download is PORTABILITY, not a backup tier — no scheduling, no status surface, +> point-in-time framing. Mechanics: four sub-pages (`/backups{,/remote,/apps,/restore}`, sections +> moved VERBATIM — `scripts/backups_split_move_check.py` gates vs df7ad37), status card (3 states, +> display-only), .fab download exit (existing exporter + staging dir + guarded stream + 1h TTL; +> traversal guard red-proven). Live-validated on drill+demo incl. a supervised import round-trip. +> **NEW FINDINGS:** **(HIGH)** containerized .fab export strands Docker-VOLUME tars on the guest +> host (`docker run -v ` → host path) — bundle ships empty volumes, import brings +> the app up EMPTY; fix = host-visible staging + fail-loud post-export assertion. **(MEDIUM, +> agent)** legacy-boot PVE (LVM root, no ESP mount) → SystemDisks empty → sysKnown=false → drive +> wizard offers ZERO candidates ever. Follow-ups: .fab browser-upload; mega-zip parked. > **2026-07-13 — v0.123.0: POLISH BATCH (take-two F-15/F-11 + rename + zero-toggle). MinAgent > 0.81.0 unchanged; floor 0.122 unchanged. Requires hub v0.52.0 for F-15 (old hub = clean no-op).** diff --git a/REPORT.md b/REPORT.md index 2d452c1..4f3d2b7 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,64 +1,86 @@ -# REPORT — v0.123.0: polish batch (F-15 instant reset codes, F-11 inline confirms, "Távoli mentés" rename, zero-toggle honesty) +# REPORT — v0.124.0: backups IA restructure (four sub-pages, Felhom-offsite status card, .fab browser download) -**Date:** 2026-07-13 · **Version:** controller v0.123.0 (from v0.122.0) · **MinAgent:** 0.81.0 (UNCHANGED) · **Companion:** hub v0.52.0 (F-15 response; old hub = clean no-op) · **Floor:** 0.122 UNCHANGED (no raise in this task) +**Date:** 2026-07-13 · **Version:** controller v0.124.0 (from v0.123.0) · **MinAgent:** 0.81.0 (UNCHANGED) · **Floor:** untouched (rides the next train) · Controller-only. -## What shipped (commits `b4e0a19`, `40b5304`, `8987ce0`, `a00afcc`, `b482860`) +## What shipped (commits `0ece2ba`, `aa967fb`, `753cd83`) -1. **F-15 — reset code works the moment the email lands** (`internal/web/claim.go`). The - reset-request POST now reads the hub's response (`claim {code_hash, generation, issued_at}`, - hub ≥0.52.0) and applies it through the SAME generation-guarded consumer as the report ACK - (`report.ClaimSync.Reconcile` — one consumer, two feeds). Replayed/older generations can never - downgrade the active hash; a legacy hub's bare `{"status":"ok"}` is a no-op. -2. **F-11 — inline confirms** (`layout.html` helper + 7 call sites). `felhomConfirm(el, q, onYes)` - swaps the trigger IN PLACE to "kérdés + Igen/Mégse"; form buttons opt in with `data-confirm` - (delegated listener + `requestSubmit`). Zero native `confirm()`/`prompt()` remain — - enforced by the new committed gate `controller/scripts/native_confirm_gate.py`. The deploy - stale-data delete keeps its double acknowledgement (two chained inline steps). -3. **Tier-3 rename** — customer-facing "NAS-mentés" → **"Távoli mentés"** across `backups.html`, - `offbox_handlers.go` flashes, and the `backup/offbox.go` quota copy; the manual-target form is - generalized to any SFTP target. Executed as a count-guarded python sweep (multibyte rule); - enforced by `controller/scripts/offbox_rename_gate.py`. "Hálózati tárhely" (NAS network - storage) is a different feature and was deliberately untouched. -4. **Zero-toggle honesty** — configured+escrowed offbox with no app selected: muted hint - "Nincs távoli mentésre jelölt alkalmazás — jelölj ki legalább egyet." + the run reports - "Sikeres — nincs mentésre jelölt alkalmazás" (LastWarning; status stays ok). -5. **Dev-box green-gate fix** (`backup.go` `atomicPromoteTar`): fsync via O_RDWR — read-only - fsync is refused on Windows; the two F7 atomic-dump tests were permanently red on the dev box - since CAMPAIGN-3 (Linux unchanged). +1. **IA split** — `/backups` (Áttekintés), `/backups/remote` (Távoli mentés), `/backups/apps` + (Alkalmazások), `/backups/restore` (Visszaállítás), sidebar children (Tárhely pattern). + Sections MOVED verbatim from the v0.123.0 page — the committed one-shot + `scripts/backups_split_move_check.py` verifies all 15 blocks against baseline `df7ad37` + (whitespace-normalized; the two allowed divergences — tier-3 anchor retarget + the + restore-to-verify relocation — are encoded explicitly). Shared handler builders extracted + (`backupsCommonData`/`backupsOffboxData`); flash redirects, tier-3 anchors and the + tier2-config back-link retargeted per page. +2. **Felhom-offsite status card** — three honest states from local data, display-only by + construction (no form/button inside the card; unit-enforced). Decision 2 honored: the card + never changes anything. +3. **.fab browser download** — existing export pipeline + staging dest + guarded streaming exit + with estimate-first UX, post-stream cleanup and a 1h TTL sweep; per-app sequential batch; + portability framing everywhere (decision 3). Import stays drive-scan. -## Tests + red-proofs (all run → FAIL → restored) +## Tests + red-proofs (run → FAIL → restored) -- `TestResetRequest_ResponseHashAppliedImmediately` (+ generation-guard + legacy-hub no-op - tests). **Red-proof:** response handling disabled → FAIL `gen=1 hash-is-new=false`. -- `native_confirm_gate.py` **red-proof:** one `onsubmit="return confirm('redproof')"` - reintroduced → exit 1; restored → 0. -- `offbox_rename_gate.py` **red-proof:** one 'A NAS-mentés…' flash reintroduced → exit 1. -- `TestOffbox_ZeroToggledRunReportsEmptiness` (+ covered-run negative). **Red-proof:** emptiness - branch disabled → FAIL `LastWarning = ""`. -- Full green gate (build+vet+test, all packages) + template_id/emoji/native_confirm/offbox_rename - gates green before every commit. +- **Move check red-proof:** one moved block reworded → exit 1 ("1 block(s) rewritten"); restored → 0. +- **Traversal guard red-proof:** guard loosened to raw prefix-matching → `../decoy.fab` came back + **200 serving the decoy** — the test caught it (got 200, want 400). Restored → green. +- **Round-trip + corruption (data-mutation red-proof):** real `executeExport` → bundle → + real `executeImport` → byte-identical restored content; a mid-file-corrupted copy is REFUSED + (job fails, app not started, nothing restored). Integrity = gzip CRC + manifest segment + validation — there is NO per-file checksum (documented; the spec's "manifest checksum" wording + adapted to reality). +- Nav/state tests (marker per section, present on exactly one page; `.Page` ids), status-card + state tests, TTL sweep test (fake mtimes/now). Full green gate + all five template gates green + per commit. -## Live validation (drill box qm300/guest 9201 + demo 9201, controller 0.123.0 both, healthy) +## §13 live validation -- **F-15 exact-failure-path re-run:** "Visszaállító kód kérése" on felhom.enkisfelhom.hu/claim → - controller log `claim-sync: hub claim code cached (generation 3)` **1 second** after the POST → - Viktor entered the emailed code: **accepted immediately, first try** (take-two failure closed). -- **F-11 automation-completes proof:** offbox restore-to-verify driven END-TO-END by browser - automation — inline strip rendered, Igen → restore ran ("A távoli visszaállítás elindult…", - banner "Távoli visszaállítás folyamatban: actualbudget…"), zero freezing. -- **Rename:** /backups renders with ZERO 'NAS-mentés' strings (screenshots in session record); - only the allowlisted "saját NAS vagy Felhom offsite tárhely" intro mention remains. -- **Zero-toggle:** untoggled ActualBudget → hint appeared → run → "Sikeres — nincs mentésre - jelölt alkalmazás" + status Rendben → re-toggled (box left as found). +- **Deployed:** 0.124.0 on the drill guest (qm300) AND demo 9201, both healthy. +- **Four-page walk (drill, screenshots in session record):** every moved control clicked once — + tier-2 Beállítás (+ retargeted back-link), tier-3 toggle (off→on), restore-to-verify from its + NEW home (`/backups/restore` flash), manual-target form open/close. Old `/backups` bookmark → + Áttekintés on both boxes; sidebar active states correct. +- **Status card:** state 3 (no card) and state 2 ("Aktív — nincs kijelölt alkalmazás") both + live-rendered on the drill box; state 1 by unit render test (demo has a managed target — spec + says do NOT unconfigure). +- **.fab download:** drill ActualBudget — estimate 3.6 KB shown first, bundle 2102 B streamed, + staging bundle removed (server log). Demo ActualBudget — downloaded to the operator machine, + sha256-identical after placement on `felhom-usb/exports` (`e2a1233f…` both ends). +- **Supervised import (Viktor GO):** ON DEMO (see finding 2 for why not drill): scan found the + bundle, manifest read (exported-by 0.124.0), "Importálás kész! Az alkalmazás sikeresen + visszaállítva.", ActualBudget Up (healthy). **The round-trip surfaced finding 1.** + +## FINDINGS (both pre-existing, surfaced by this task's first real containerized round-trip) + +1. **(HIGH — data-loss trap) Containerized .fab export ships EMPTY volume data.** + `exportVolumeData` runs `docker run -v :/out alpine tar …` where `` is the + controller-CONTAINER's `os.MkdirTemp` path — the guest's docker daemon resolves it against the + HOST filesystem, so the tar lands stranded on the guest host (`/tmp/felhom-export-*/…`, found + with ActualBudget's 66 KB tar in it) and the bundle's `data/volumes` is empty while the + manifest claims volume data and the export reports SUCCESS. Import then wipes the app's + volumes and brings it up EMPTY. Affects every golden/bootstrap (containerized) box for + volume-backed apps; HDD-data apps are unaffected (direct FS copy). Demo's ActualBudget was + repaired from the stranded tar; both temp dir and staged bundle cleaned. **Fix direction:** + stage the docker-run output under a host-visible bind (e.g. the controller data volume) or use + `docker cp`/stdout streaming; plus a post-export assertion that manifest-claimed volume tars + exist in the bundle (fail LOUD). +2. **(MEDIUM — agent-side, out of this controller-only task) Drive wizard offers zero candidates + on a legacy-boot PVE.** With LVM root and no ESP mount, every system mount resolves to + device-mapper, `storage.SystemDisks` finds no raw disk → `sysKnown=false` → `isSystemBacked` + fail-safes EVERYTHING to system → `ListCandidateDisks` is always empty (drill box; a hot-added + blank 5G disk stayed invisible). The fail-safe is right to exist but overreaches here — needs + an agent ruling (e.g. resolve device-mapper parents to their PVs' disks). ## Observations -- The queued **customer-target coexistence** task (`managed_by` on OffboxTarget) remains a - SEPARATE task with a pending operator fork — not started, per spec. -- NAS-string inventory beyond backups.html: the two customer-facing quota strings in - `internal/backup/offbox.go` (renamed here); everything else is the distinct "Hálózati tárhely" - feature or English comments/tests (untouched by design). -- Windows-only pre-existing failure found+fixed: read-only-handle fsync in `atomicPromoteTar` - (the F7 tests were red on the dev box, green on Linux — a silent hole in the local green gate). -- Browser-automation note: `find`-ref clicks on plain submit buttons focused but did not submit - (coordinate clicks did); the data-confirm buttons worked via refs. Cosmetic, tooling-side. +- **.fab browser-UPLOAD** remains the noted follow-up (import is drive-scan only — downloaded + bundles return via a drive/share, exactly as exercised here). +- **Combined mega-zip** stays parked (decision; per-app sequential batch shipped instead). +- **appexport packages LIVE state** (fresh config copy + DB dump + live data), NOT the tier-1 + recovery unit — §8's "stale unit" concern is moot; no "futtass előbb mentést" needed. +- The import page's drive-label mojibake ("TÃ˘rhely (felhom-usb)") is the KNOWN open fix-5 item. +- `NoUserDataBackupWarning` is referenced by the per-app section but never set by any handler + (dead conditional, pre-existing); `DBDumpTotalBytes` was computed but never rendered — dropped + in the split. +- Drill box left as found: hot-added disk detached + LV removed; ActualBudget re-toggled; demo + exports dir + stray export temp cleaned. diff --git a/REUSE.md b/REUSE.md index c404ac2..32010b5 100644 --- a/REUSE.md +++ b/REUSE.md @@ -208,6 +208,8 @@ Cross-repo edges: - **New background job**: `sched.Every`/`sched.Daily` registration block in controller/cmd/controller/main.go. - **New template function**: `Server.templateFuncMap` (controller/internal/web/funcmap.go) — obey v2 state-suffix vocabulary. - **New page/nav item**: `baseData` + sidebar in controller/internal/web/templates/ (nested sub-links pattern `.nav-links-nested`); must pass `controller/scripts/template_id_gate.py` + `controller/scripts/emoji_gate.py` + `controller/scripts/native_confirm_gate.py` + `controller/scripts/offbox_rename_gate.py`. +- **Guarded file download (v0.124.0)**: `handler_export_download.go` — the canonical shape for streaming a server-side file to the browser: accept a BASENAME only (shape regexp + no separators/`..`), `filepath.Join` then assert `filepath.Dir(path) == dir`, `io.Copy` (never ReadAll), `Content-Disposition: attachment`, remove after a successful stream, TTL sweep (`sweepFabDownloads(dir, now, maxAge, logger)` — now injected for tests). Red-proof the guard by loosening to prefix-matching (the `..` case must fail). +- **Backups sub-page data**: `backupsCommonData(page, title, r)` + `backupsOffboxData(data)` (handlers.go) — the ONLY builders for the four `/backups*` pages; a new backups section extends these, never re-derives in a page handler. Section moves are gated by `scripts/backups_split_move_check.py` (one-shot vs df7ad37). - **Consequential-action confirm (LIGHT)**: `felhomConfirm(el, question, onYes)` in layout.html (v0.123.0) — the trigger swaps in place to "kérdés + Igen/Mégse"; form buttons opt in with `data-confirm="…"` (delegated listener, `requestSubmit` keeps formaction/name-value). NEVER native `confirm()`/`prompt()` (OS-modals freeze browser automation — drill F-11; `native_confirm_gate.py` enforces). Heavy destructive flows keep the `.confirm-overlay` `openDialog` pattern. - **New hub event**: typed `Notify*` wrapper on Notifier + hub allowlist entry (cross-repo). - **New app integration**: `integrations.Manager.RegisterHandler` with `IntegrationKey(provider, target)`. diff --git a/controller/README.md b/controller/README.md index 9edb3b1..801eb1e 100644 --- a/controller/README.md +++ b/controller/README.md @@ -343,9 +343,11 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte **Architecture:** `internal/appexport/` package with `ExportStackProvider` adapter interface (same pattern as `backup.StackDataProvider`). `exportAdapter` in `main.go` bridges `stacks.Manager` to the provider. -**API endpoints:** `/api/export/estimate`, `/api/export/start`, `/api/export/status`, `/api/export/bundles`, `/api/export/manifest`, `/api/export/import`, `/api/export/import/status`. +**API endpoints:** `/api/export/estimate`, `/api/export/start`, `/api/export/status`, `/api/export/bundles`, `/api/export/manifest`, `/api/export/import`, `/api/export/import/status`; **browser download (v0.124.0):** `/api/export/download/estimate`, `/api/export/download/start`, `GET /api/export/download?file=`. -**UI:** Export button on app info page, standalone import page at `/import` accessible from the stacks page header. +**Browser download (v0.124.0 — portability, NOT a backup tier):** the same export pipeline runs with dest = `/fab-downloads/` (same producer → byte-identical bundle), then streams via a guarded endpoint (basename-shape + dir-containment guard; `Content-Disposition: attachment`; `io.Copy`; the staged bundle is removed after the stream and a 1h TTL sweep runs on startup + each start). Estimate is shown BEFORE starting; the batch UI downloads apps one at a time (no combined archive). Import remains drive-scan only — a downloaded bundle returns via a drive/share. `handler_export_download.go`. + +**UI:** Export button on app info page, the "Hordozható mentéscsomag (.fab)" section on `/backups/restore` (per-app download + batch), standalone import page at `/import`. --- @@ -354,6 +356,19 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete, self-sufficient backup** — any single tier can fully restore an app. +**Page map (v0.124.0 IA split — sidebar children under Biztonsági mentés):** + +| Route | Page | Sections | +|-------|------|----------| +| `/backups` | Áttekintés | storage overview, whole-guest Rendszermentés, status stat cards, single-copy warning | +| `/backups/remote` | Távoli mentés | Felhom-offsite status card (3 states, display-only), tier-3 status block + quota, participation toggles (+ zero-toggle hint), manual-target form (`#offbox-section`) | +| `/backups/apps` | Alkalmazások | schedule, Adatbázisok table, per-app 1./2./3. tier rows (tier-2 config entry; tier-3 actions deep-link to `/backups/remote#offbox-section`) | +| `/backups/restore` | Visszaállítás | restore panel, offbox restore-to-verify list, .fab download/import loop | + +Shared data builders: `backupsCommonData` (chrome + full-status + flash) + `backupsOffboxData` +(offbox target/toggles) in `handlers.go`; shared partials in `templates/backups_shared.html`. +The split is MOVE-only vs v0.123.0 — `scripts/backups_split_move_check.py` gates it. + | Tier | Contents | Location | Can fully restore? | |------|----------|----------|--------------------| | **1. Nightly restic** | DB + Config + User data | Same drive as app | Yes (not against drive failure) |