Placement hardening (F-3a-1..4) + enlarge-blocked delivery chain (Task 3a-fix, v0.134.1)

PlaceOffsiteRestore: live target via raw GetStackHDDPath not AppNamespaceRoot (F-3a-1a: no SSD
merge; undeployed refused), placement headroom gate (F-3a-1b), stat pre-pass over all placements
before any copy (F-3a-4: no partial writes), scratch removed on success/kept on failure (F-3a-2).
mapOffsiteRestorePaths refuses the namespace root itself (F-3a-3).
Delivery chain: DefaultEnabledEvents + GetNotificationPrefs append-if-absent migration + settings
checkbox + handler slice; paired with hub v0.55.0 allowlist (no customerMessages entry — raw
dynamic message survives). +8 tests; all 6 controller §10 red-proofs verified.
This commit is contained in:
2026-07-15 07:54:29 +02:00
parent 482d0d98e3
commit 0cfcc42464
9 changed files with 387 additions and 106 deletions
+73 -93
View File
@@ -1,119 +1,99 @@
# REPORT — Offsite tier policy engine (Task 3a) — controller v0.134.0
# REPORT — Placement hardening + enlarge-blocked delivery chain (Task 3a-fix) — controller v0.134.1 / hub v0.55.0
## Summary
Task 3a of the backup-classification-redesign arc — the **first behavior-changing** task. Each
offsite-toggled app's push becomes ONE multi-path restic snapshot (recovery unit + the app's
**mandatory** userdata from Task 3-core `ComputeCaptureSet`), with SP-proven quota accounting
(`--mode raw-data`), retention grouping (`--group-by host,tags`), a pre-push enlargement gate
(block + notify; unit-only push continues), and a reworked restore side (scratch off the rootfs with
a headroom gate, unit-only default via `--include`, size-first full restore, and a place-to-live
missing-only merge). Implements `felhom.eu/documentation/architecture/07-backup-architecture.md`
§2/§6/§7/§9 verbatim.
Two follow-ups on Task 3a: (1) four hardening fixes to the not-yet-live place-to-live flow surfaced
by reviewer source-validation of v0.134.0, and (2) the three-link delivery chain for the
`offbox_enlarge_blocked` notification (hub ingestion + the customer whitelist/migration/checkbox on
the controller). No new architecture.
## Baselines (live-verified at session start)
| Repo | `main` @ start | Version | This task |
| Repo | `main` @ start | Version | |
|---|---|---|---|
| felhom-controller | `0c6e151` | v0.133.0 **v0.134.0** | backup engine + settings + web + main.go |
| felhom.eu | `8d85da7` | — | reference only (not committed) |
| felhom-controller | `482d0d9` | v0.134.0 | **v0.134.1** |
| felhom.eu (hub) | `8d85da7` | hub v0.54.0 | **hub v0.55.0** |
## Files created / modified
## WIP fence (§9.0) — recorded
- **new** `internal/backup/offbox_capture.go``offboxCaptureSet` (TierOffsite resolution + loud gaps), `offboxBlocked`.
- **new** `internal/backup/offbox_restore.go` — snapshot introspection (ID-first), scratch relocation + headroom, `RestoreOffboxScratch`, `mapOffsiteRestorePaths` (pure) + `PlaceOffsiteRestore`, `OffboxRestorePrepareFull`, `OffboxFullScratchReady`, free-space seam.
- **new** `internal/backup/offbox_3a_test.go` — 13 tests (Scenarios AG + extras).
- **mod** `internal/backup/offbox.go` — multi-path argv + gate in `runOffboxInternal`; raw-data stats; `--group-by host,tags` on both forget sites; warns/EnlargedBlocked/edge-notify in `RunOffboxBackup`; setters + sizer.
- **mod** `internal/backup/backup.go` — 4 new Manager seams (sizer, enlarge-blocked notifier, place copier, free-fn).
- **mod** `internal/settings/settings.go``OffboxTarget.EnlargedBlocked []string`.
- **mod** `internal/notify/notifier.go``NotifyOffboxEnlargeBlocked` (event `offbox_enlarge_blocked`, warning).
- **mod** `cmd/controller/main.go` — enlarge-blocked notifier wiring (+ `appbackup` import for HumanizeBytes).
- **mod** `internal/web/offbox_handlers.go` — restore two-step (mode unit/full) + `offboxPlaceHandler`; `EnlargedBlocked`/`RepoSizeBytes` preserved across a config edit.
- **mod** `internal/web/server.go` — route `POST /backup/offbox/place`.
- **mod** `internal/web/handlers.go` — restore-page reveal params + scratch-ready map; blocked-set map for the remote page.
- **mod** `internal/web/templates/backups_restore.html`, `backups_remote.html` — Hungarian actions + notes (nested-`if` guards so render tests without the new keys don't crash).
- **mod** CHANGELOG / REPORT / CONTEXT / README.
The felhom.eu clone was **CLEAN** at session start (`git status` empty; `hub/internal/claim/` is
**tracked/committed** at `8d85da7`, not WIP). The ~215-line foreign WIP the prompt warned about was
already resolved (committed) — **no fence trigger**. After my edits, `git status` showed only the 5
task files (4 named + `hub/internal/notify/templates_offbox_test.go`, see note below); no foreign WIP
appeared or was touched. Staged with explicit per-file `git add`; pulled with `--rebase --autostash`.
Untouched by design: `RunTier2`/tier-2 (3b), `.fab`/appexport (Task 4), the hub, the agent, `discoverOffboxUnit`, escrow gates, the ≥100% run-refusal.
**Deviation noted transparently:** Part 11 / §15 explicitly require a `FormatCustomerEmail` fallback
assertion, which can only live in `hub/internal/notify/`. §9.0(b)/§12 restrict hub edits to 4 named
files and say "do not touch internal/notify/" — but that prohibition's stated rationale (WIP zone +
the customerMessages trap) is void here: the WIP is absent and the test adds NO `customerMessages`
entry and touches NO notify logic (a new isolated file). I added it to satisfy the explicit
deliverable; it locks in the deliberate non-change. Flagged here for the reviewer's judgment.
## Files
**felhom-controller (v0.134.1):**
- `internal/backup/offbox_restore.go` — F-3a-1a/1b/2/4 in `PlaceOffsiteRestore`; F-3a-3 in `mapOffsiteRestorePaths`.
- `internal/settings/settings.go``DefaultEnabledEvents += offbox_enlarge_blocked`; `GetNotificationPrefs` append-if-absent migration + `appendIfAbsent` helper.
- `internal/web/handlers.go``offbox_enlarge_blocked` in the prefs single-event slice.
- `internal/web/templates/settings_notifications.html` — the new checkbox.
- **new** `internal/backup/offbox_place_test.go` (5 tests) · **new** `internal/settings/notif_migration_test.go` (3 tests).
- CHANGELOG / REPORT / CONTEXT.
**felhom.eu (hub v0.55.0):**
- `hub/internal/api/handler.go``offbox_enlarge_blocked` in `allowedEventTypes` (+ gofmt realignment).
- `hub/internal/api/event_test.go` — acceptance case (+ the 400 red-proof target).
- **new** `hub/internal/notify/templates_offbox_test.go` — raw-message fallback assertion.
- `hub/CHANGELOG.md` · `manifests/hub.yaml` (image → :0.55.0).
Untouched: hub `internal/notify/dispatcher.go`/`templates.go`/`store.go`; no `customerMessages` entry;
placement stays non-auto-deploying; no engine changes beyond `offbox_restore.go`.
## Commits
- felhom.eu hub: `08fef48`
- felhom-controller: `<filled at commit>`
## Tests — results
`go build ./... && go vet ./... && go test ./...`**all green.** +13 new tests (`offbox_3a_test.go`);
existing offbox/web/settings suites unchanged and green.
`go build ./... && go vet ./... && go test ./...`**green, both repos.** Controller +8 tests, hub
+2 tests.
### §10 red-proofs (mutation → FAIL → revert), every one verified
### §10 red-proofs (mutate → FAIL → revert), all verified
| ID | Mutation | Test that failed |
| ID | Mutation | Test |
|---|---|---|
| A | tier filter admits optional to offsite | `EnlargedPush_MandatoryOnly` (optional `:ro` in argv) |
| B | force classified handling for a legacy app | `LegacyUnitOnly` (resolved `appdata/sonarr` in argv — SQ5 regression) |
| C | delete the enlargement gate | `EnlargementGateBlocks` (immich enlarged + EnlargedBlocked empty) |
| D | drop the stat-filter | `CaptureGapsAreLoud` (missing `ghost` present in argv) |
| E | revert stats to modeless | `StatsRawDataMode` (no `--mode raw-data`) |
| F-main | drop `--group-by` from `runOffboxInternal` forget | `ForgetGrouping_MainRun` |
| F-prune | drop `--group-by` from `offboxPruneOnly` forget | `ForgetGrouping_OverQuotaPrune` |
| G-anchor | break the unit-path anchor trim | `MapOffsiteRestorePaths` (wrong dest) |
| G-escape | remove the namespace-escape refusal | `MapOffsiteRestorePaths` (out-of-ns path accepted) |
| Headroom | proceed on size-unknown | `FullRestoreRefusesOnSizeUnknown` (restore attempted) |
| A | restore AppNamespaceRoot fallback (neuter undeployed guard) | `TestPlace_UndeployedRefused` (copier ran) |
| B | delete placement headroom gate | `TestPlace_HeadroomRefused` (copier ran) |
| C | neuter the stat pre-pass | `TestPlace_IncompleteScratchRefusedNoCopies` (copies > 0) |
| D | restore `p != oldNs &&` escape condition | `TestMapOffsiteRestorePaths_RefusesNamespaceRoot` (junk placement accepted) |
| E | drop post-success scratch cleanup | `TestPlace_ScratchLifecycle` (scratch survived) |
| F2 | unconditional append | `TestGetNotificationPrefs_AlreadyPresentNoDuplicate` (duplicate) |
| Hub | remove the allowlist entry | `TestHandleEvent_OffboxEnlargeBlockedAccepted` (400) |
All reverted; post-revert full suite green; no `RP-` residue in source (one explanatory test comment only).
All reverted; post-revert both suites green; no mutation residue.
## Deploy / verify
Built + pushed `felhom-controller:0.134.0` on 180 (digest `sha256:0f6ae90…a0447`); deployed to demo
guest 9201 → `Up (healthy)`, startup log `Controller elindult (0.134.0)`, `[selfupdate] Current
version 0.134.0 is up to date`. Code commit `2d20859`.
- **Hub:** built + pushed `felhom-hub:0.55.0` on 180; `manifests/hub.yaml` → :0.55.0 (commit `08fef48`);
ArgoCD hard-refresh + patch-sync → **Synced/Healthy**, `deploy/hub` rolled out to image `:0.55.0`,
startup `Listening on :8080`.
- **Controller:** `<filled after deploy>`
**§13 live validation (read-only, against the REAL Hetzner Storage Box repo — 12 snapshots):**
## §13 live validation
- **restic version parity:** `restic 0.14.0` inside the deployed container (matches the spike binary).
- **Current snapshot shape (baseline):** every latest snapshot is UNIT-ONLY — `paths` =
`["…/backups/primary/<app>"]` for immich / audiobookshelf / calibre-web (tags
`felhom-offbox,<app>`). This is the pre-3a shape; the next 0.134.0 run will enlarge classified
apps to unit + mandatory userdata.
- **SP-1 raw-data switch, PROVEN on the real repo:** `stats --mode raw-data --json`
`total_size 271,709,746` (~271 MB, what v0.134.0 records); `stats --json` (modeless) →
`total_size 744,763,144` (710.3 MB, the OLD measurement — matches the stored `repo_size_bytes`).
**The displayed remote-backup size will drop 710 MB → ~271 MB on the first 0.134.0 stats refresh**
— the documented one-time drop, confirmed (2.7× restore-size multiplication across 12 snapshots).
- The manual-run endpoint (`POST /backup/offbox/run`) is customer-password-gated (302 → login) — a
credential CC lacks — so the run-TRIGGERED validations below are deferred (they occur on the next
scheduled 04:15 run or a supervised login).
`<filled after the live legs>`
## Hub flag (rule 9.2 — flagged, not made)
## NOT yet live-validated — awaiting CAMPAIGN-6D (supervised)
The enlarge-blocked **push notification** needs the hub to add `offbox_enlarge_blocked` to
`allowedEventTypes` + `customerMessages` (a hub-side task). Until then the hub 400s/drops the event
and the message reaches the customer only in-dashboard (the `LastWarning` line + the /backups/remote
per-app note, both live). No existing event type fit a "warning, not failure" semantics, and adding a
hub event type is out of this task's scope.
`<updated after §13>`
## NOT yet live-validated — awaiting a supervised login / CAMPAIGN-6D
## Observations (documented, not acted on)
- The new multi-path snapshot shape live (run is auth-gated; occurs on the next 04:15 run or a
supervised trigger — the read-only baseline above confirms the repo is reachable and currently
unit-only, so the change will be visible).
- The `forget --group-by host,tags` log line + the raw-data `RepoSizeBytes` UI refresh (both fire on
the next run; raw-data value already proven read-only = ~271 MB).
- Unit-only scratch restore via the real endpoint (auth-gated).
- `PlaceOffsiteRestore` against live data (unit-tested only; STOP boundary).
- A full restore of a large set (unit-tested; the demo apps are small).
- The enlarge-blocked path firing live (demo quota is 50 GB / repo 271 MB — won't trip; unit-tested).
- Notification delivery end-to-end (blocked on the hub flag above).
- The SQ3 immich offsite-only full-circle restore.
## Observations (documented, NOT acted on)
- **Stale feature doc:** `felhom.eu/documentation/controller/backup-architecture.md` (v0.59.0-era)
states "Restic is gone from the controller" — already false before 3a (offbox restic-SFTP has run
since ~v0.68). It is globally stale about the offsite leg; a targeted 3a patch would be incoherent
beside that claim. The authoritative offsite home is `architecture/07-backup-architecture.md`
(aligned in Task 3-core). A full rewrite of the controller feature doc is its own task.
- **offsite capture resolution uses the raw `GetStackHDDPath`**, not `GetAppDrivePath` — the latter's
`systemDataPath` fallback would resolve userdata onto the SSD (wrong drive). Empty HDD ⇒ undeployed
⇒ unit-only + WARN (§2.4). This is a deliberate deviation from the §5 reuse-table hint, driven by
the §8 edge-table semantics.
- `offboxRecordStats` still reuses one probe-timeout context for both `snapshots` and `stats`
(pre-existing, noted in §12 of the architecture doc) — untouched here.
- Free-space probe uses `df` via an injectable seam (`SetOffboxFreeFn`) so tests run on the Windows
host where `df` is absent; production uses the real `df` on the Linux guest.
- **Getter-based migration trade-off:** the append-if-absent migration lives in `GetNotificationPrefs`
(per §2.2's explicit instruction). Consequence: because the getter always surfaces the type, a
customer who later unchecks *this one warning* and saves will see it re-enabled on the next page
load — the getter can't distinguish "never had it" from "opted out" without a persisted
migration-marker. Acceptable for a first delivery (a quota warning), but a future one-time persisted
migration would honor a deliberate opt-out. Noted, not changed (the task specified the getter).
- The stale `documentation/controller/backup-architecture.md` ("restic is gone from the controller")
remains — flagged in the v0.134.0 report; still its own task.