From 0f311adaa13e6c3abea3cb0f8b2872eb363bd0e9 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 12 Jul 2026 09:18:05 +0200 Subject: [PATCH] =?UTF-8?q?controller=20v0.118.0:=20backup=20integrity=20F?= =?UTF-8?q?7/F6/F5=20=E2=80=94=20docs=20+=20CHANGELOG=20+=20REPORT=20+=20C?= =?UTF-8?q?ONTEXT?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit F7 atomic volume dumps, F6 no single-copy (volume-only tier-2 + honest single-drive signal), F5 stale-primary sweep. Part 4 locality fork = A (doc-only). Live-validated on 9201: F7 money-shot byte-identical through mid-write cut, F6/F5 confirmed, restore round-trip byte-identical. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf --- CHANGELOG.md | 33 +++++++++++++++ CONTEXT.md | 18 ++++++++- REPORT.md | 96 ++++++++++++++++++++++---------------------- controller/README.md | 27 +++++++++++++ 4 files changed, 126 insertions(+), 48 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 96f9554..b2729e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,38 @@ ## Changelog +### v0.118.0 — backup integrity: atomic volume dumps (F7) + no single-copy (F6) + stale-primary sweep (F5) (2026-07-12) — MinAgent: 0.81.0 + +Fixes CAMPAIGN-3 (`felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`) backup findings. MinAgent +UNCHANGED (0.81.0) — all changes are controller-local; no new agent API consumed. + +- **F7 (HIGH) — atomic volume dumps.** `backup.DumpAppVolumes` now writes the tar to `.tar.tmp`, + fsyncs it, and only atomically `os.Rename`s it over the restore point on success — the same + crash-safe pattern the DB-dump path already uses (`appbackup/dbdump.go` DumpOne), extended with a + best-effort directory fsync. Before this, tar wrote the `.tar` IN PLACE, so a mid-write NFS cut left + a 0-byte tar REPLACING the last good dump (tier-1 restore is replace-semantics → an empty volume). + Now any tar error / timeout / dead-NFS EIO removes ONLY the `.tmp`; the last good `.tar` is + byte-untouched. The `.tar.tmp` name (ends `.tmp`, not `.tar`) is invisible to the + restore-point/stale scans; orphan `.tar.tmp` from a killed run is swept. New `tarVolume` test seam. +- **F6 (LOW) — no single-copy backups.** Volume-only apps (no HDD_PATH, backups on sys_drive) now flow + through the tier-2 cross-drive copy (`RunAllTier2` no longer skips non-HDD apps) — a second copy on + the secondary drive (the 3-2-1 intent). Their restore-point drive label is no longer blank (clear + "Belső SSD (rendszer)"). A single-drive box (no off-drive target) surfaces an HONEST + `SingleCopyWarning` banner on the backup page instead of implying a 3-2-1 guarantee it cannot keep. +- **F5 (LOW) — stale primary-dir sweep.** After each backup cycle, `pruneStalePrimaryDirs` removes an + orphaned `backups/primary/` dir an app left on an OLD drive when its HDD_PATH moved (invisible + disk residue). LOAD-BEARING GUARDS: removes only when the app is deployed AND its current namespace + root differs from the dir's drive; NEVER touches the app's current-drive dir (the live restore + point) or an undeployed app's dir; only ever operates strictly under a `backups/primary/` prefix. +- **Part 4 (operator fork) — backup-target locality: option A (keep locality), document-only.** NAS + apps' tier-1 artifacts stay beside the data on the NAS; tier-2's cross-drive copy is the off-NAS + leg. Documented plainly (backup feature doc) so the NAS-outage window is never a surprise; no code + change (option B, retarget-to-local, was not selected). +- **Live-validated (demo 9201):** F7 money-shot — a mid-write `exportfs -u` during a volume dump left + all 5 nas-media volume tars BYTE-IDENTICAL (sha unchanged), no 0-byte, no leftover `.tar.tmp`, run + `success:false`; next run produced fresh good tars. F6 — actualbudget/seerr now on + felhom-usb/secondary. F5 — a seeded stale dir on the wrong drive swept, current dirs kept. Restore + round-trip byte-identical. Tests incl. the F7 truncation red-proof + F5 guard red-proofs, all green. + ### v0.117.0 — consuming-namespace NAS verification + deploy-view truth (RCA fixes 2+4) (2026-07-11) — MinAgent: 0.81.0 Controller half of the RCA fix pair (agent v0.84.0 ReassertNetworkMounts). Source: diff --git a/CONTEXT.md b/CONTEXT.md index 308d24d..6eaf855 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,23 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-11 (v0.116.1 — observability pass; pairs with agent v0.83.0 + hub v0.46.0) +Last updated: 2026-07-12 (v0.118.0 — backup integrity F7/F6/F5; MinAgent 0.81.0 unchanged) + +> **2026-07-12 — v0.118.0: BACKUP INTEGRITY (LIVE on 9201). MinAgent 0.81.0 unchanged.** Fixes +> CAMPAIGN-3 backup findings (`documentation/audits/CAMPAIGN-3-2026-07-11.md`). **F7 (HIGH) atomic +> volume dumps:** `DumpAppVolumes` writes `.tar.tmp` → fsync → `os.Rename` over the `.tar` only on +> success (`atomicPromoteTar`), mirroring dbdump.go DumpOne; a mid-write NFS cut can no longer +> truncate the last good tar to 0 bytes. **F6 (LOW) no single-copy:** `RunAllTier2` no longer skips +> volume-only apps (they now get a cross-drive tier-2 copy); sys_drive restore-point label is clear +> ("Belső SSD (rendszer)"); single-drive box shows an honest `SingleCopyWarning` banner. **F5 (LOW) +> stale-primary sweep:** `pruneStalePrimaryDirs` removes an orphaned `backups/primary/` dir on an +> OLD drive after an HDD_PATH move (guarded: deployed + different-current-drive only, never a restore +> point). **Part 4 locality fork → operator chose (A) keep locality, doc-only** (NAS tier-1 stays on +> the NAS; tier-2 is the off-NAS leg). LIVE: F7 money-shot (all NAS tars byte-identical through a +> mid-write cut, no 0-byte, success:false); F6 (actualbudget/seerr on felhom-usb/secondary); F5 +> (seeded stale dir swept, current kept); restore round-trip byte-identical. Seams: `tarVolume`, +> `perAppTier2`. Task C (F8/F4) + Task D (ring/alerting) still queued; Peti reaches 0.118 + agent 0.85 +> at his next train (agentless-on-proxmox2 gap noted). > **2026-07-11 — v0.116.0/0.116.1: OBSERVABILITY PASS (LIVE on 9201; agent v0.83.0 + hub v0.46.0). > MinAgent: 0.81.0 unchanged.** The debug ring (`LogBuffer`) now ALWAYS exists — logger = diff --git a/REPORT.md b/REPORT.md index 29c6ca8..e11a833 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,58 +1,60 @@ -# REPORT — v0.117.0: consuming-namespace NAS verification + deploy-view truth (RCA fixes 2+4) +# REPORT — v0.118.0: backup integrity (F7 atomic dumps · F6 no-single-copy · F5 stale sweep) -**Date:** 2026-07-11 · **Version:** controller v0.117.0 (from v0.116.1) · **MinAgent:** 0.81.0 -UNCHANGED (all checks controller-namespace-local; no new agent API) · **Deployed:** guest 9201 -(`0.117.0` healthy) · **NOT published/vouched** — Peti stays 0.113.0. Pairs with agent v0.84.0 -(RCA fix 1). Commits: `c0f3e12` + `4a9c54a` (linux-suite test pin). -**Source:** `felhom.eu/documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md`. +**Date:** 2026-07-12 · **Version:** controller v0.118.0 (from v0.117.0) · **MinAgent:** 0.81.0 (UNCHANGED) +· **Deployed:** guest 9201 (`0.118.0` healthy) · **Source:** `felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`. ## What shipped -- **`internal/system/fsclass*.go`** — statfs f_type classifier for THIS process's namespace: - `network` (nfs/cifs/smb2) | `autofs` (healthy idle trigger, NEVER force-mounted) | `stub` | - `unknown` (error/3 s timeout → fail open). Seams: `statfsFn`, `netProbeFSClass`, - `Server.classifyFSPath`, `Router.classifyFSPath`. -- **Fix 2a — probe assertion:** `--netprobe` child creates the probe file first (legitimately - triggers the automount), then REQUIRES a mounted network fs → exit 5 → category - `not_network_fs`, new §3.2 Hungarian message, full rollback, nothing registered. -- **Fix 2b — deploy refusal:** `POST /api/stacks/{name}/deploy` → `refuseNetworkStubDeploy`: a - registered network `HDD_PATH` classifying as stub refuses 409; idle autofs / live / unknown / - local / unregistered / empty all proceed. -- **Fix 2c — stub badge:** `networkStorageWarnings` → (warnings, stubs); stub leg runs even when - the agent is unreachable; dashboard + stacks cards render "Hálózati tárhely hibás — az alkalmazás - nem a NAS-t látja"; stub WINS over unreachable; the unreachable line stays byte-identical - (template-asserted). Pure core `networkStorageWarningsIn` + `stubNetworkPaths`. -- **Fix 4 — deploy-view truth (the RCA S-C symptom):** deployed-app storage select marks `selected` - by the STORED `HDD_PATH`; absent stored path → extra disabled ` (nem elérhető)` option; - `IsDefault` selects only for NEW deploys. +- **F7 (HIGH) — atomic volume dumps.** `backup.DumpAppVolumes` writes the tar to `.tar.tmp`, + fsyncs it, then atomically `os.Rename`s over the restore point ONLY on success (+ best-effort dir + fsync via `atomicPromoteTar`) — mirroring the DB-dump path (`dbdump.go` DumpOne). Any tar + error / timeout / dead-NFS EIO removes only the `.tmp`; the last good `.tar` is byte-untouched. The + `.tar.tmp` is invisible to restore-point/stale scans (ends `.tmp`, not `.tar`); orphan tmps are + swept. New `tarVolume` seam for the red-proof. +- **F6 (LOW) — no single-copy.** `RunAllTier2` no longer skips non-HDD apps → volume-only apps get a + cross-drive tier-2 copy. sys_drive restore-point label is now clear ("Belső SSD (rendszer)"), never + blank. `SingleCopyWarning` (honest HU banner) shows on a single-drive box with no off-drive target. +- **F5 (LOW) — stale primary sweep.** `pruneStalePrimaryDirs` (end of each backup cycle) removes an + orphaned `backups/primary/` dir on an OLD drive after an HDD_PATH move. Guards: deployed + + different-current-drive only; never the current-drive dir or an undeployed app's dir; strictly under + a `backups/primary/` prefix. +- **Part 4 — locality fork: operator chose (A) keep locality, document-only.** No code change; the + NAS-outage behaviour + tier-2 mitigation are documented plainly. -## Tests + red-proofs +## Tests + red-proofs (all green) -| Test | Result | Red-proof | -|---|---|---| -| magic table + seam + real-local-dir-is-stub (linux) | PASS | — | -| `TestNetProbeChild_StubRefused` / `_AutofsAfterCreateRefused` / `_RealLocalDirRefused` (linux) | PASS | assertion disabled → **the stub VERIFIED** (exit 0) → FAIL | -| `TestNetAdd_StubProbe_RollsBackNotRegistered` (rollback + not registered + Hungarian) | PASS | same mutation → job phase `done` → FAIL | -| `TestRefuseNetworkStubDeploy_Table` (7 rows) | PASS | mounted-only gate → idle-autofs + unknown rows FAIL (over-refusal) | -| `TestNetworkStorageWarningsIn_StubWins` / `_UnreachableAloneUnchanged` | PASS | — | -| `TestNetworkBadgeTemplates_DistinctStrings` (unreachable line byte-identical) | PASS | — | -| `TestDeployPage_DeployedSelectShowsStoredHDDPath` (real Manager + real template render) | PASS | IsDefault-only revert → default drive selected → FAIL (both assertions) | -| `TestDeployPage_MissingStoredPathRendersTruthOption` | PASS | — | +- F7: happy atomic promote (no `.tmp` left); **truncation red-proof** — pre-seeded good `.tar`, tar + fails mid-write → original byte-unchanged, no 0-byte (revert to in-place write → fails); leftover + `.tar.tmp` swept. +- F6: volume-only app in the tier-2 run set (revert the skip → fails); single-drive honest signal; + sys_drive drive-label non-blank. +- F5: redeployed residue removed; **guard red-proofs** — undeployed app's restore point kept, + current-drive dir kept. +- Existing `TestListRestorePoints_DriveLabel/SSD` updated (blank → clear label, the F6 behaviour). -Full suite green on Windows AND on 180 (linux); the linux run caught the pre-existing C5 child-body -suite needing the classifier pinned to `network` (fixed in `4a9c54a` — the real statfs correctly -refuses a tempdir). Gates: `template_id_gate.py` + `emoji_gate.py` OK. +## Live validation (demo 9201, real pipeline, sim-NAS rails — exportfs only) -## Deploy + live validation +- **F7 money-shot:** established good tars (radarr sha `847a97ad…` +4 others), then `exportfs -u` the + NFS target 4 s into a backup during volume dumps → run `success:false`; **all 5 nas-media volume + tars BYTE-IDENTICAL** (clean host-side diff), **no 0-byte file, no leftover `.tar.tmp`**; the next + clean run produced fresh good tars (`success:true`). This is the exact campaign F7 repro, now safe. +- **F6:** after a tier-2 cycle, volume-only `actualbudget` + `seerr` appear under + `felhom-usb/backups/secondary/` (previously none); `actualbudget` snapshot `drive_label` = + "Belső SSD (rendszer)"; single-copy banner correctly absent on the multi-drive demo. +- **F5:** a seeded stale `nas-media/backups/primary/actualbudget` (actualbudget lives on sys_drive) was + swept by the next backup; the real sys_drive dir + radarr's nas-media dir were kept. +- **Restore round-trip:** calibre-web tamper → restore → BYTE-IDENTICAL (53 files), container healthy — + F7's write change did not affect restore. -- Image `0.117.0` built+pushed on 180, deployed to 9201 via the bootstrap mechanism; healthy. -- **Q1c reboot validation (operator GO, with agent 0.84.0):** post-reboot, no manual action — - CWA auto-started healthy on the NAS, health `status=ok` on the first cycle, dashboard renders NO - stub badge (grep count 0 against the live page). -- **NOT live-validated (test-level only, by design):** the stub refusal branches — producing a live - stub post-fix would require breaking a healthy guest (the agent now heals it at start). Covered by - the red-proven unit/render tests above. +## NOT live-validated / standing items -## Follow-ups +- Single-drive `SingleCopyWarning` live (the demo has enrolled drives) — unit-tested only. +- Task C (F8/F4) and Task D (ring/alerting) remain queued. +- Peti's box (controller 0.113 / agentless-on-proxmox2) reaches 0.118 + agent 0.85 at his next train — + noted for the roadmap; the agentless-on-proxmox2 cluster gap stands. +- No publish/floor movement; agent untouched (MinAgent 0.81.0). -RCA fixes 3/5/6 queued as separate tasks. Peti publish train: 0.84.0 + 0.117.0. +## Box state at wrap + +controller 0.118.0 healthy on 9201; all backups consistent (fresh tier-1 + tier-2 incl. volume-only +apps); NAS re-exported; no stale test residue. diff --git a/controller/README.md b/controller/README.md index 5f7be2b..ec850b1 100644 --- a/controller/README.md +++ b/controller/README.md @@ -363,6 +363,13 @@ self-sufficient backup** — any single tier can fully restore an app. No tier depends on another tier's data. - **Tier 2 is configurable for ALL apps** — not just apps with HDD data. Non-HDD apps back up config + DB dumps to the secondary drive (small but protects against drive failure). + **(v0.118.0, F6):** the nightly tier-2 RUN (`RunAllTier2`) now actually includes volume-only apps + too — it used to skip every non-HDD app, leaving them a single controller-level copy on `sys_drive`. + A sys_drive app's restore-point drive label is now clear ("Belső SSD (rendszer)"), never blank. +- **3-2-1 on single vs multi drive (v0.118.0, F6):** on a box with a second physical drive, tier-2 is + the off-drive copy. On a **single-drive** box (no off-drive target at all) there is genuinely only + ONE local copy — `FullBackupStatus.SingleCopyWarning` surfaces an honest Hungarian notice on the + backup page ("Csak egy másolat készül…") instead of implying a 3-2-1 guarantee the box cannot keep. - The `AppBackupPrefs.Enabled` field in settings.json is legacy and not read by any code. **Per-app Tier 2 contents by app type:** @@ -380,6 +387,26 @@ self-sufficient backup** — any single tier can fully restore an app. The nightly backup has two phases that run sequentially. All paths are **per-drive** — each physical drive gets its own restic repo and per-app DB dump directories. +> **Atomic dump writes (v0.118.0, CAMPAIGN-3 F7).** BOTH dump paths are crash-safe: the DB dump +> (`dbdump.go` DumpOne) and the Docker-volume dump (`DumpAppVolumes`) write to a `.tmp` sibling, fsync, +> then `os.Rename` over the restore point ONLY on success. A mid-write failure (a NFS cut mid-tar, an +> EIO, a timeout) removes only the `.tmp` and leaves the last good `.sql`/`.tar` **byte-untouched** — +> a tier-1 restore is replace-semantics, so an in-place write that got truncated to 0 bytes used to +> destroy the only restore point. `.tar.tmp` files are invisible to the restore-point/stale scans and +> orphans are swept on the next run. + +> **Stale-primary sweep (v0.118.0, F5).** After each cycle, `pruneStalePrimaryDirs` removes an +> orphaned `backups/primary/` dir left on an OLD drive when an app's `HDD_PATH` moved to another +> drive. Guarded: only for a DEPLOYED app whose CURRENT drive differs from the dir's drive; never the +> current-drive dir (the live restore point) or an undeployed app's dir; strictly under `backups/primary/`. + +> **NAS backup locality (v0.118.0, CAMPAIGN-3 Part 4 — decision A).** A NAS-resident app's tier-1 +> artifacts live **on the NAS itself** (`nas-media/backups/primary/`), beside the data. During a +> NAS outage both the app data AND its freshest tier-1 dump are on the dead device — the **tier-2** +> cross-drive copy to a local drive is the off-NAS leg that saves them (and only after it has run). +> This locality is deliberate (kept in the fork over retargeting tier-1 to a local drive); the tier-2 +> copy is the mitigation. Stated here so the outage window is never a surprise. + **Drive layout (v0.26.0):** ``` /