Tier-2 engine rework: class-driven legs, v2 layout, NAS-target exclusion (Task 3b, v0.135.0)

tier2_capture.go: classified apps get TierSecondary per-bind legs (paperless copy shrinks — export
drops); legacy apps keep the byte-identical resolver set. v2 relpath-mirroring layout
(backups/secondary/<stack>/{marker LAST, recovery-unit/, hdd/<rel>/, userdata/<rel>/}); N>1 native
(errTier2MultiDir/tier2AppDataName deleted). Migration=delete-and-rebuild + reconcile; all RemoveAll
via tier2SafeRemove (refuses outside backups/secondary/). SSD=state-only tier. selectTier2Target
never picks network storage (pinned+auto, F-6C-1). Restore reads v2 behind a marker gate.
Part 0: offbox_enlarge_blocked is a persisted one-time Load seed (opt-out sticks), not a getter
append. Part 0.5: offsite restore scratch prefers a local (non-network) path.
Full v2 test suite + all 10 §10 red-proofs verified. Destructive writes bounded to backups/secondary/.
This commit is contained in:
2026-07-15 10:10:20 +02:00
parent 1245a6c46e
commit 3603d1fc7f
20 changed files with 1014 additions and 484 deletions
+45 -113
View File
@@ -1,143 +1,75 @@
# REPORT — Placement hardening + enlarge-blocked delivery chain (Task 3a-fix) — controller v0.134.1 / hub v0.55.0
# REPORT — Tier-2 engine rework (Task 3b) — controller v0.135.0
## Summary
Two follow-ups on Task 3a: (1) four hardening fixes to the not-yet-live place-to-live flow surfaced
by reviewer source-validation of v0.134.0, and (2) the three-link delivery chain for the
`offbox_enlarge_blocked` notification (hub ingestion + the customer whitelist/migration/checkbox on
the controller). No new architecture.
Task 3b: the tier-2 column of the matrix (architecture §2/§8). The tier-2 appdata leg becomes
**class-driven** for classified apps (per-bind mandatory + optional; paperless's copy shrinks as
`export` drops out), legacy apps keep a byte-identical resolver set — both mapped into a new **v2
relpath-mirroring layout** that lifts the N>1 refusal structurally and gains a reconcile step; the SSD
fallback is an enforced **state-only** tier; and `selectTier2Target` gains the F-6C-1 **network-target
exclusion** (pinned + auto). Restore reads v2 behind a marker gate. Plus **Part 0** (persisted prefs
seed — fixes the 3a-fix un-disableable checkbox) and **Part 0.5** (offsite scratch prefers local).
Every destructive write lands only under `backups/secondary/<stack>/` (code-asserted).
## Baselines (live-verified at session start)
| Repo | `main` @ start | Version | → |
|---|---|---|---|
| felhom-controller | `482d0d9` | v0.134.0 | **v0.134.1** |
| felhom.eu (hub) | `8d85da7` | hub v0.54.0 | **hub v0.55.0** |
## WIP fence (§9.0) — recorded
The felhom.eu clone was **CLEAN** at session start (`git status` empty; `hub/internal/claim/` is
**tracked/committed** at `8d85da7`, not WIP). The ~215-line foreign WIP the prompt warned about was
already resolved (committed) — **no fence trigger**. After my edits, `git status` showed only the 5
task files (4 named + `hub/internal/notify/templates_offbox_test.go`, see note below); no foreign WIP
appeared or was touched. Staged with explicit per-file `git add`; pulled with `--rebase --autostash`.
**Deviation noted transparently:** Part 11 / §15 explicitly require a `FormatCustomerEmail` fallback
assertion, which can only live in `hub/internal/notify/`. §9.0(b)/§12 restrict hub edits to 4 named
files and say "do not touch internal/notify/" — but that prohibition's stated rationale (WIP zone +
the customerMessages trap) is void here: the WIP is absent and the test adds NO `customerMessages`
entry and touches NO notify logic (a new isolated file). I added it to satisfy the explicit
deliverable; it locks in the deliberate non-change. Flagged here for the reviewer's judgment.
| felhom-controller | `1245a6c` | v0.134.1 | **v0.135.0** |
## Files
**felhom-controller (v0.134.1):**
- `internal/backup/offbox_restore.go` — F-3a-1a/1b/2/4 in `PlaceOffsiteRestore`; F-3a-3 in `mapOffsiteRestorePaths`.
- `internal/settings/settings.go``DefaultEnabledEvents += offbox_enlarge_blocked`; `GetNotificationPrefs` append-if-absent migration + `appendIfAbsent` helper.
- `internal/web/handlers.go``offbox_enlarge_blocked` in the prefs single-event slice.
- `internal/web/templates/settings_notifications.html` — the new checkbox.
- **new** `internal/backup/offbox_place_test.go` (5 tests) · **new** `internal/settings/notif_migration_test.go` (3 tests).
- CHANGELOG / REPORT / CONTEXT.
- **new** `internal/backup/tier2_capture.go``tier2CaptureSet` (TierSecondary legs + gaps), `tier2Leg`, `tier2DestRel`.
- **mod** `internal/backup/tier2.go``selectTier2Target` (network exclusion, two sizes, `StateOnly`), `RunTier2` v2 body (migration/mirror/reconcile/marker-last), `tier2SafeRemove` + `tier2Reconcile` + `classifyTier2Rel`, `Tier2Info` rewired, header + `RunTier2` comments truthful; deleted `errTier2MultiDir`/`tier2AppDataName`.
- **mod** `internal/backup/tier2_restore.go` — v2 marker gate + two-subtree missing-only merge; deleted `errTier2MultiDirRestore`.
- **mod** `internal/backup/offbox_restore.go` — Part 0.5 scratch local-preference.
- **mod** `internal/backup/backup.go``tier2SSDFits` seam.
- **mod** `internal/backup/appbackup_bridge.go``ClassMandatory/Optional/Excluded` aliases.
- **mod** `internal/settings/settings.go` — Part 0 seed (`OffboxEnlargeNoticeSeeded` + `seedOffboxEnlargeNotice` at Load; getter append removed); `CrossDriveBackup.LastWarning`.
- **mod** `internal/web/handlers.go` + `templates/backups_apps.html` — surface `Tier2LastWarning`.
- **tests** new `tier2_v2_test.go`, `offbox_scratch_test.go`; rewrote `notif_migration_test.go` (Part 0), `tier2_restore_test.go` fixture (v2), `async_restore_test.go` fixture (v2), `tier2_test.go` signatures; **deleted** obsolete `tier2_appdata_test.go`.
- CHANGELOG / REPORT / CONTEXT / README.
**felhom.eu (hub v0.55.0):**
- `hub/internal/api/handler.go``offbox_enlarge_blocked` in `allowedEventTypes` (+ gofmt realignment).
- `hub/internal/api/event_test.go` — acceptance case (+ the 400 red-proof target).
- **new** `hub/internal/notify/templates_offbox_test.go` — raw-message fallback assertion.
- `hub/CHANGELOG.md` · `manifests/hub.yaml` (image → :0.55.0).
Untouched: hub `internal/notify/dispatcher.go`/`templates.go`/`store.go`; no `customerMessages` entry;
placement stays non-auto-deploying; no engine changes beyond `offbox_restore.go`.
## Commits
- felhom.eu hub: `08fef48`
- felhom-controller: `0cfcc42`
Untouched: offbox engine (except `offboxRestoreScratchDir`), appexport, hub, agent; `rsyncMirror`/`rsyncRestoreMissing` unchanged.
## Tests — results
`go build ./... && go vet ./... && go test ./...`**green, both repos.** Controller +8 tests, hub
+2 tests.
`go build ./... && go vet ./... && go test ./...`**green.** New: v2 suite (§7 AH + reconcile
keep/remove + safe-remove boundary proof + `classifyTier2Rel` table), Part 0 (idempotent seed +
opt-out-sticks + fresh-customer), Part 0.5 (local-preference + network-only WARN).
### §10 red-proofs (mutate → FAIL → revert), all verified
| ID | Mutation | Test |
|---|---|---|
| A | restore AppNamespaceRoot fallback (neuter undeployed guard) | `TestPlace_UndeployedRefused` (copier ran) |
| B | delete placement headroom gate | `TestPlace_HeadroomRefused` (copier ran) |
| C | neuter the stat pre-pass | `TestPlace_IncompleteScratchRefusedNoCopies` (copies > 0) |
| D | restore `p != oldNs &&` escape condition | `TestMapOffsiteRestorePaths_RefusesNamespaceRoot` (junk placement accepted) |
| E | drop post-success scratch cleanup | `TestPlace_ScratchLifecycle` (scratch survived) |
| F2 | unconditional append | `TestGetNotificationPrefs_AlreadyPresentNoDuplicate` (duplicate) |
| Hub | remove the allowlist entry | `TestHandleEvent_OffboxEnlargeBlockedAccepted` (400) |
| A | classified app routed to the legacy resolver | `ClassifiedPaperless` (`export` appears in dest) |
| B | legacy routed through ComputeCaptureSet | `LegacyMultiDir` (empty legs) |
| C | marker written before the legs | `MigrationAndMarkerLast` (marker present after a leg failure) |
| D-reconcile | disable stale detection | `Reconcile` (stale dir survives) |
| D-boundary | neuter the safe-remove prefix assert | `Reconcile` (removal outside destBase not refused) |
| E | SSD copies optional (no state-only filter) | `SSDStateOnly` (optional on the SSD) |
| F1 | drop the pinned IsNetwork check | `NetworkExclusion/pinned` (NAS pin honored) |
| F2 | drop the auto IsNetwork check (NFS iterated first) | `NetworkExclusion/auto` (NAS selected) |
| G2 | neuter the restore marker gate | `RestoreRefusesOldLayout` (merge runs on old layout) |
| Part 0 | don't persist the seed marker | `SeedOffboxEnlargeNotice_OptOutSticks` (type re-enabled) |
All reverted; post-revert both suites green; no mutation residue.
All reverted; post-revert full suite green; no residue.
## Deploy / verify
- **Hub:** built + pushed `felhom-hub:0.55.0` on 180; `manifests/hub.yaml` → :0.55.0 (commit `08fef48`);
ArgoCD hard-refresh + patch-sync → **Synced/Healthy**, `deploy/hub` rolled out to image `:0.55.0`,
startup `Listening on :8080`.
- **Controller:** built + pushed `felhom-controller:0.134.1` on 180; deployed to guest 9201 →
`Up (healthy)`. Commit `0cfcc42`.
`<filled after deploy>`
## §13 live validation — ALL LEGS PASS
## §13 live validation
The dashboard session was established with `180:~/.config/credentials` `C4_PASSWORD` (the value is
**single-quoted** in the file — stripping the quotes is required; an earlier run that kept the quotes
produced a false `Hibás jelszó`, since corrected). Login → 302 + `felhom_session` cookie; the flow ran
entirely on 180 (LAN → dashboard) so the password was never transferred or echoed.
`<filled after the live legs>`
- **Leg 2 — manual run (LIVE):** POST `/backup/offbox/run` → 302; run log:
`backed up calibre-web (…, 1 mandatory path(s))` (enlarged shape), `audiobookshelf`/`immich`
`not deployed — offsite push is unit-only` (the §2.4 undeployed WARN firing live), `backup OK: 3
app(s) backed up, 15 snapshot(s), 49s`. **Leg 2a** confirmed by `snapshots --no-lock --json`:
calibre-web newest = `[…/backups/primary/calibre-web, …/userdata/media/books]` (unit + mandatory).
- **Leg 2b — raw-data quota (LIVE):** stored `repo_size_bytes = 280,932,890` (267.9 MB) == live
`stats --mode raw-data` — down from 744 MB modeless pre-3a.
- **Leg 2c — forget grouping:** both call sites carry `--group-by host,tags` (deployed + unit-tested);
a successful forget is logged SILENTLY by design (only failures log), so no live line — the ~40s gap
before "backup OK" + the stable 15-snapshot retention evidence it ran.
- **Leg 3 — unit-only scratch restore (LIVE):** POST `mode=unit` → scratch landed on the DATA DRIVE
`/mnt/felhom-drives/nas-media/backups/offsite-restore/calibre-web` (NOT the rootfs — F-A1); the
reconstructed tree contains ONLY `…/backups/primary/calibre-web/{compose,volume-dumps}` (unit-only
scope, `--include` working); rootfs `DataDir/offbox-restore` absent.
- **Leg 4 — full restore two-step (LIVE):** step 1 `mode=full` → 302 to
`/backups/restore?full_prep=calibre-web&full_size=258.3+MB` (size shown BEFORE start); step 2
`mode=full&confirm=1` → the scratch now also holds `…/userdata/media/books/.calnotes/…` (whole
snapshot, SP-3.1 abs-path reconstruction).
- **Leg 5 — prefs round-trip (LIVE):** `/settings/notifications` renders `event_offbox_enlarge_blocked`
**checked** (migration surfaced it enabled); a save round-trip → the type **survived** (F3
checkbox-drop trap avoided); the hub logged `Notification preferences updated for demo-felhom:
events=[… offbox_enlarge_blocked …]` — the migrated type reached the hub whitelist end-to-end.
- **Leg 6 — delivery (LIVE, one synthetic event):** pushed `offbox_enlarge_blocked`/warning through the
real controller→hub path → **hub_status 200** (was 400 before v0.55.0); hub logged
`Event from demo-felhom: offbox_enlarge_blocked (warning)` + `Operator email sent for
demo-felhom/offbox_enlarge_blocked`. The message was the debug endpoint's generic text (it fixes the
body); the dynamic two-number message survival is unit-proven (`templates_offbox_test.go`) — no
`customerMessages` entry, so `FormatCustomerEmail` falls back to the raw message.
- **Leg 7 — hygiene:** the calibre-web scratch (legs 34) removed; 180 session temp files removed;
read-only inspection scripts removed from the container. `PlaceOffsiteRestore` was NOT run (6D).
## 6D-pending
## NOT yet live-validated — awaiting CAMPAIGN-6D (supervised)
`<updated after §13>`
- `PlaceOffsiteRestore` against live data (the STOP boundary — merges into live app data).
- Organic enlarge-block firing (demo repo 268 MB / quota 50 GB won't trip; unit-tested + the delivery
chain proven live via the synthetic event).
- The SQ3 immich offsite-only full-circle restore-and-boot.
## Observations
## Observations (documented, not acted on)
- **Getter-based migration trade-off:** the append-if-absent migration lives in `GetNotificationPrefs`
(per §2.2's explicit instruction). Consequence: because the getter always surfaces the type, a
customer who later unchecks *this one warning* and saves will see it re-enabled on the next page
load — the getter can't distinguish "never had it" from "opted out" without a persisted
migration-marker. Acceptable for a first delivery (a quota warning), but a future one-time persisted
migration would honor a deliberate opt-out. Noted, not changed (the task specified the getter).
- The stale `documentation/controller/backup-architecture.md` ("restic is gone from the controller")
remains — flagged in the v0.134.0 report; still its own task.
- **Demo-state touches during §13 (benign, noted):** the leg-5 prefs save persisted the demo
customer's `enabled_events` to explicitly include `offbox_enlarge_blocked` (previously getter-migrated
— idempotent, no behavior change); leg 6 sent one operator test email. Legs 34 scratch dirs were
removed (leg 7). No live app data was modified (`PlaceOffsiteRestore` not run).
- **Credential note:** `C4_PASSWORD` in `180:~/.config/credentials` is **single-quoted** — strip the
quotes before use (an unstripped value yields a false `Hibás jelszó`). The credential is valid.
- The `Tier2LastWarning` surfacing is minimal (one line on the app's tier-2 row); the SSD state-only
hint was already rendered via `EffectiveIsSSD`/`EffectiveDesc` (no new template needed there).
- `tier2SSDFits` seam added because `system.GetDiskUsage` is Linux-only (nil on the Windows test host)
— production uses the real predicate.