Offsite tier policy engine: mandatory userdata, raw-data quota, restore rework (Task 3a, v0.134.0)
Each toggled app's offsite push = one multi-path restic snapshot (recovery unit + TierOffsite mandatory userdata via ComputeCaptureSet); legacy/undeployed stay unit-only. Loud capture gaps (SP-3.4: restic 0.14.0 silently skips missing paths). Quota = stats --mode raw-data (SP-1; displayed size drops once). Pre-push enlargement gate blocks the userdata enlargement over-quota (unit-only push continues; EnlargedBlocked; edge-triggered notify). forget --group-by host,tags on both sites (SP-2). Restore reworked: scratch off the rootfs + headroom gate (F-A1), unit-only default via --include, size-first full, place-to-live missing-only merge (never --delete). UI: unit/full-two-step/place actions + per-app blocked note; route POST /backup/offbox/place. HUB FLAG: offbox_enlarge_blocked event needs hub allowlist for push delivery. +13 tests; all 10 §10 red-proofs verified. No tier-2/.fab/hub/agent changes.
This commit is contained in:
@@ -1,101 +1,98 @@
|
||||
# REPORT — Capture-set computation (INERT; Task 3-core) — controller v0.133.0
|
||||
# REPORT — Offsite tier policy engine (Task 3a) — controller v0.134.0
|
||||
|
||||
## Summary
|
||||
|
||||
Task 3-core of the backup-classification-redesign arc
|
||||
(`felhom.eu/documentation/architecture/07-backup-architecture.md` §3; tier×class matrix §2; spike
|
||||
verdicts in `SPIKE-restic-snapshot-shape-2026-07-14.md`). Ships **one pure function**,
|
||||
`appbackup.ComputeCaptureSet`, that turns an app's classified binds + a tier + the app's live
|
||||
`hddPath` into the tier-filtered, structurally-guarded, containment-deduped absolute capture set that
|
||||
the 3a (offsite) and 3b (tier-2) engines will consume — plus a `Skipped` list for structurally unsafe
|
||||
would-be captures and a pure `CrossAppOverlaps` advisory. **Deliberately INERT** like Task 2: no
|
||||
backup tier changes behavior; nothing consumes any of it yet.
|
||||
Task 3a of the backup-classification-redesign arc — the **first behavior-changing** task. Each
|
||||
offsite-toggled app's push becomes ONE multi-path restic snapshot (recovery unit + the app's
|
||||
**mandatory** userdata from Task 3-core `ComputeCaptureSet`), with SP-proven quota accounting
|
||||
(`--mode raw-data`), retention grouping (`--group-by host,tags`), a pre-push enlargement gate
|
||||
(block + notify; unit-only push continues), and a reworked restore side (scratch off the rootfs with
|
||||
a headroom gate, unit-only default via `--include`, size-first full restore, and a place-to-live
|
||||
missing-only merge). Implements `felhom.eu/documentation/architecture/07-backup-architecture.md`
|
||||
§2/§6/§7/§9 verbatim.
|
||||
|
||||
## Baselines (live-verified at session start)
|
||||
|
||||
| Repo | `main` @ start | Version | This task |
|
||||
|---|---|---|---|
|
||||
| felhom-controller | `95f3180` | v0.132.0 → **v0.133.0** | `appbackup` engine + `stacks` wiring test |
|
||||
| felhom.eu | `b279312` | — | §3 docs alignment only (commit `8d85da7`) |
|
||||
| felhom-controller | `0c6e151` | v0.133.0 → **v0.134.0** | backup engine + settings + web + main.go |
|
||||
| felhom.eu | `8d85da7` | — | reference only (not committed) |
|
||||
|
||||
## Files created / modified
|
||||
|
||||
- **new** `controller/internal/appbackup/captureset.go` — `ComputeCaptureSet`, `CaptureTier`,
|
||||
`CapturePath`, `SkippedPath`, `CaptureSet`, `CrossAppOverlaps`, `Overlap`; the §8 pipeline +
|
||||
structural guards.
|
||||
- **new** `controller/internal/appbackup/captureset_test.go` — Groups A–F (7 tests).
|
||||
- **new** `controller/internal/stacks/captureset_wiring_test.go` — Group G, F-S3 no-seam end-to-end.
|
||||
- **mod** `controller/CHANGELOG.md` (v0.133.0, newest-on-top), `controller/REPORT.md` (this),
|
||||
`controller/CONTEXT.md`, `controller/README.md` (appbackup surface, one block).
|
||||
- **mod** `felhom.eu/documentation/architecture/07-backup-architecture.md` §3 (as-built API sketch;
|
||||
separate commit `8d85da7`).
|
||||
- **new** `internal/backup/offbox_capture.go` — `offboxCaptureSet` (TierOffsite resolution + loud gaps), `offboxBlocked`.
|
||||
- **new** `internal/backup/offbox_restore.go` — snapshot introspection (ID-first), scratch relocation + headroom, `RestoreOffboxScratch`, `mapOffsiteRestorePaths` (pure) + `PlaceOffsiteRestore`, `OffboxRestorePrepareFull`, `OffboxFullScratchReady`, free-space seam.
|
||||
- **new** `internal/backup/offbox_3a_test.go` — 13 tests (Scenarios A–G + extras).
|
||||
- **mod** `internal/backup/offbox.go` — multi-path argv + gate in `runOffboxInternal`; raw-data stats; `--group-by host,tags` on both forget sites; warns/EnlargedBlocked/edge-notify in `RunOffboxBackup`; setters + sizer.
|
||||
- **mod** `internal/backup/backup.go` — 4 new Manager seams (sizer, enlarge-blocked notifier, place copier, free-fn).
|
||||
- **mod** `internal/settings/settings.go` — `OffboxTarget.EnlargedBlocked []string`.
|
||||
- **mod** `internal/notify/notifier.go` — `NotifyOffboxEnlargeBlocked` (event `offbox_enlarge_blocked`, warning).
|
||||
- **mod** `cmd/controller/main.go` — enlarge-blocked notifier wiring (+ `appbackup` import for HumanizeBytes).
|
||||
- **mod** `internal/web/offbox_handlers.go` — restore two-step (mode unit/full) + `offboxPlaceHandler`; `EnlargedBlocked`/`RepoSizeBytes` preserved across a config edit.
|
||||
- **mod** `internal/web/server.go` — route `POST /backup/offbox/place`.
|
||||
- **mod** `internal/web/handlers.go` — restore-page reveal params + scratch-ready map; blocked-set map for the remote page.
|
||||
- **mod** `internal/web/templates/backups_restore.html`, `backups_remote.html` — Hungarian actions + notes (nested-`if` guards so render tests without the new keys don't crash).
|
||||
- **mod** CHANGELOG / REPORT / CONTEXT / README.
|
||||
|
||||
No engine edits (`RunTier2`/`RunOffboxBackup`/`RestoreOffbox` untouched), no web/scheduler wiring, no
|
||||
`ClassifyBinds`/`ValidateBackupSpec`/`ParseComposeClassifiableBinds` change, no `AppDataDirNames` call.
|
||||
|
||||
## Design (as-built)
|
||||
|
||||
Fixed pipeline (§8): **legacy short-circuit → tier filter (§2) → structural guards → equal-Abs
|
||||
collapse (mandatory > optional) → containment dedup (keep ancestor) → sort by Abs.** Pure: no
|
||||
`os`/`exec`/`filepath`/logging; slash algebra throughout (`RelPath` is forward-slash, resolved Abs is
|
||||
an in-container Linux path — `filepath` on the Windows test host would flip separators and break the
|
||||
containment prefix checks). Resolution: `RootHDD → path.Join(hddPath, rel)`,
|
||||
`RootUserdata → path.Join(hddPath, "userdata", rel)`. Structural guards are load-bearing security: the
|
||||
compose parser path.Cleans but does not reject `..`, so an unlisted writable `${HDD_PATH}/../x` bind
|
||||
reaches the function classed mandatory; the guard moves it to `Skipped` (with the bare-drive-root and
|
||||
reserved-`backups/` guards) instead of into a captured path.
|
||||
Untouched by design: `RunTier2`/tier-2 (3b), `.fab`/appexport (Task 4), the hub, the agent, `discoverOffboxUnit`, escrow gates, the ≥100% run-refusal.
|
||||
|
||||
## Tests — results
|
||||
|
||||
`go build ./... && go vet ./... && go test ./...` — **all green, both repos** (felhom.eu has no Go).
|
||||
|
||||
New tests (8 total): `internal/appbackup` +7 (`PerTierSplit`, `LegacyInert`, `ExcludedInvisible`,
|
||||
`StructuralGuards`, `LegitDotDotName`, `ContainmentAndCollision`, `CrossAppOverlaps`);
|
||||
`internal/stacks` +1 (`CaptureSet_Wiring`). Full-suite package count unchanged, all `ok`.
|
||||
`go build ./... && go vet ./... && go test ./...` — **all green.** +13 new tests (`offbox_3a_test.go`);
|
||||
existing offbox/web/settings suites unchanged and green.
|
||||
|
||||
### §10 red-proofs (mutation → FAIL → revert), every one verified
|
||||
|
||||
| ID | Mutation | Test that must fail | Observed failure |
|
||||
|---|---|---|---|
|
||||
| **B** (SQ5) | legacy short-circuit resolves binds as mandatory | `LegacyInert` | legacy app resolved `[appdata/sonarr, media/tv]` into Paths (+HasClassification=true) |
|
||||
| **A** (tier) | `TierOffsite` includes optional | `PerTierSplit` | offsite Paths gained `/userdata/media/photos` |
|
||||
| **D** (guard) | traversal guard deleted | `StructuralGuards` | `/mnt/evil` (escaped root) present in Paths; 2 Skipped not 3 |
|
||||
| **E1** (contain) | containment dedup disabled | `ContainmentAndCollision` | descendant `appdata/paperless/media` not dropped |
|
||||
| **E2** (mand-wins) | mandatory strength = optional | `ContainmentAndCollision` | collapsed `/userdata/media` class degraded to optional |
|
||||
| **G** (wiring) | tier constants swapped in the filter | `CaptureSet_Wiring` (end-to-end) | real-Manager secondary lost photos / offsite gained it — no fake absorbed the typo |
|
||||
| ID | Mutation | Test that failed |
|
||||
|---|---|---|
|
||||
| A | tier filter admits optional to offsite | `EnlargedPush_MandatoryOnly` (optional `:ro` in argv) |
|
||||
| B | force classified handling for a legacy app | `LegacyUnitOnly` (resolved `appdata/sonarr` in argv — SQ5 regression) |
|
||||
| C | delete the enlargement gate | `EnlargementGateBlocks` (immich enlarged + EnlargedBlocked empty) |
|
||||
| D | drop the stat-filter | `CaptureGapsAreLoud` (missing `ghost` present in argv) |
|
||||
| E | revert stats to modeless | `StatsRawDataMode` (no `--mode raw-data`) |
|
||||
| F-main | drop `--group-by` from `runOffboxInternal` forget | `ForgetGrouping_MainRun` |
|
||||
| F-prune | drop `--group-by` from `offboxPruneOnly` forget | `ForgetGrouping_OverQuotaPrune` |
|
||||
| G-anchor | break the unit-path anchor trim | `MapOffsiteRestorePaths` (wrong dest) |
|
||||
| G-escape | remove the namespace-escape refusal | `MapOffsiteRestorePaths` (out-of-ns path accepted) |
|
||||
| Headroom | proceed on size-unknown | `FullRestoreRefusesOnSizeUnknown` (restore attempted) |
|
||||
|
||||
All mutations reverted; post-revert full suite green; no `RED-PROOF` residue in the new files.
|
||||
All reverted; post-revert full suite green; no `RP-` residue in source (one explanatory test comment only).
|
||||
|
||||
## Deploy / verify
|
||||
|
||||
Built + pushed `felhom-controller:0.133.0` on 180 (digest `sha256:0832106…c772d8`); deployed to demo
|
||||
guest 9201 (bootstrap-managed). Code commit `2668ac4` (controller `main`); docs `8d85da7` (felhom.eu
|
||||
`main`).
|
||||
Built + pushed `felhom-controller:0.134.0` on 180; deployed to demo guest 9201. Per-guest `docker ps`
|
||||
+ startup log + the §13 live-validation evidence recorded below.
|
||||
|
||||
- `docker ps` (guest 9201): `gitea.dooplex.hu/admin/felhom-controller:0.133.0 Up (healthy)`.
|
||||
- Startup log: `[INFO] Event pushed: controller_started (info) — Controller elindult (0.133.0)`;
|
||||
recovery-unit capture + hub report + health probes all normal (the seerr/radarr/calibre-web
|
||||
`no such host` warns are pre-existing demo-DNS noise, unrelated to this change).
|
||||
- **INERT-silence check PASS:** `grep -iE 'capturese|computecapture|crossapp'` over the container
|
||||
logs returns nothing — the package has zero call sites, so no feature log line fires at runtime.
|
||||
_Deploy + live-validation evidence appended after the live run — see the deploy commit._
|
||||
|
||||
## Live-validation scope
|
||||
## Hub flag (rule 9.2 — flagged, not made)
|
||||
|
||||
Live validation beyond deploy-health is **inherently N/A** for an inert pure package: it has no
|
||||
runtime surface, no UI, no behavior change. The real live legs belong to 3a (offsite) and 3b (tier-2)
|
||||
acceptance, which consume this function.
|
||||
The enlarge-blocked **push notification** needs the hub to add `offbox_enlarge_blocked` to
|
||||
`allowedEventTypes` + `customerMessages` (a hub-side task). Until then the hub 400s/drops the event
|
||||
and the message reaches the customer only in-dashboard (the `LastWarning` line + the /backups/remote
|
||||
per-app note, both live). No existing event type fit a "warning, not failure" semantics, and adding a
|
||||
hub event type is out of this task's scope.
|
||||
|
||||
## NOT yet live-validated — awaiting CAMPAIGN-6D (supervised)
|
||||
|
||||
- `PlaceOffsiteRestore` against live data (unit-tested only; STOP boundary).
|
||||
- A full restore of a large set (unit-tested; the demo apps are small).
|
||||
- The enlarge-blocked path firing live (demo quota is 50 GB — won't trip; unit-tested).
|
||||
- Notification delivery end-to-end (blocked on the hub flag above).
|
||||
- The SQ3 immich offsite-only full-circle restore.
|
||||
|
||||
## Observations (documented, NOT acted on)
|
||||
|
||||
- **Parser-side traversal:** `ParseComposeClassifiableBinds` / `classifyRoot` path.Clean the compose
|
||||
host token but do not reject a `..` that survives cleaning (`${HDD_PATH}/../x` → RelPath `../x`).
|
||||
This is by design per the task (the parser stays a faithful extractor; the guard lives in 3-core),
|
||||
and the structural guard here is what makes it safe. If a future task ever wants defence-in-depth,
|
||||
the parser is the second place it could live — noted, not changed.
|
||||
- `ClassifyBinds` emits legacy binds with an **empty** `Class`; `ComputeCaptureSet`'s legacy
|
||||
short-circuit means those are never resolved, but `tierKeeps` also treats an empty class as
|
||||
not-captured (defensive) — so even a future caller that skipped the short-circuit could not resolve
|
||||
a classless bind. Belt-and-suspenders, intentional.
|
||||
- The equal-Abs collapse can arise from two *spellings* of one path (`hdd:userdata/media` vs
|
||||
`userdata:media`); today no catalog app does this, but the collapse + mandatory-wins rule makes it
|
||||
safe if one ever does.
|
||||
- **Stale feature doc:** `felhom.eu/documentation/controller/backup-architecture.md` (v0.59.0-era)
|
||||
states "Restic is gone from the controller" — already false before 3a (offbox restic-SFTP has run
|
||||
since ~v0.68). It is globally stale about the offsite leg; a targeted 3a patch would be incoherent
|
||||
beside that claim. The authoritative offsite home is `architecture/07-backup-architecture.md`
|
||||
(aligned in Task 3-core). A full rewrite of the controller feature doc is its own task.
|
||||
- **offsite capture resolution uses the raw `GetStackHDDPath`**, not `GetAppDrivePath` — the latter's
|
||||
`systemDataPath` fallback would resolve userdata onto the SSD (wrong drive). Empty HDD ⇒ undeployed
|
||||
⇒ unit-only + WARN (§2.4). This is a deliberate deviation from the §5 reuse-table hint, driven by
|
||||
the §8 edge-table semantics.
|
||||
- `offboxRecordStats` still reuses one probe-timeout context for both `snapshots` and `stats`
|
||||
(pre-existing, noted in §12 of the architecture doc) — untouched here.
|
||||
- Free-space probe uses `df` via an injectable seam (`SetOffboxFreeFn`) so tests run on the Windows
|
||||
host where `df` is absent; production uses the real `df` on the Linux guest.
|
||||
|
||||
Reference in New Issue
Block a user