diff --git a/REPORT.md b/REPORT.md index c6178d9..9e420f1 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,244 +1,105 @@ -# REPORT — R-77: endpoint-drift detection, samba protected-set gate, channel log honesty (2026-07-26) +# REPORT — R-82 Slice B: one quiesce window, two backup tiers (v0.175.0) (2026-07-26) -**Overwritten** per the standing rule. Controller **v0.173.0** live on demo-felhom 9201, demo-hp 9201 -and drill-r50 9201; hub **v0.74.0** live. MinAgent unchanged; `felhom-agent` untouched. +**Overwritten** per the standing rule. Controller **v0.174.0 → v0.175.0**. +Full cross-repo arc: `felhom.eu/REPORT.md`. + +**MinAgent UNCHANGED — deliberately.** This release degrades gracefully against ANY older agent. It +does not require agent v0.97.0. --- -## 1. Baselines used +## 1. Why the controller had to change -| Repo | start | end | version | -|---|---|---|---| -| felhom-controller | head after R-75 (`c7a3a90`) | `9056f01` | `v0.172.0` → **`v0.173.0`** | -| felhom.eu | `9e94479` | `9cfa619` | hub `v0.73.1` live → **`v0.74.0`** | -| felhom-agent | `dfd5d731` | untouched | `v0.96.0` | -| app-catalog-felhom.eu | `4252121` | untouched | — | +The agent gained per-target backup tiers (R-82 Slice A): local **daily** + offsite **weekly**. But +the **controller** owns quiescing — it polls `/backup/due`, stops the app stacks, calls `POST +/backup`, and restarts them. So the multi-tier schedule has to be reconciled here, and one night a +week both tiers come due at once. ---- +Two quiesce cycles that night would mean **two app outages for one night's work** — undoing the +entire argument for weekly-over-daily, since the quiesce blip was the only real cost of the offsite +tier. -## 2. Part 0 — operational repair (supervised, done first) +## 2. The dedup rule — specified, not emergent -Its own STOP condition — *"if `fingerprint` or `token` also disagree, STOP"* — **did not trigger**: -only the address had moved on either box. - -| | demo-felhom 9201 | demo-hp 9201 | +| local due | PBS due | result | |---|---|---| -| endpoint BEFORE | `192.168.0.162:8443` | `192.168.0.87:8443` | -| endpoint AFTER | `169.254.253.1:8443` | `169.254.253.1:8443` | -| **fingerprint agrees** | **true** | **true** | -| **token agrees** | **true** | **true** | -| backup | `/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml.pre-r77.bak` | same path | -| `[channel]` lines in 90 s after restart | **0** | **0** | -| hub `agent_channel_*` events since | **0** | **0** | +| yes | no | one quiesce, local backup | +| no | yes | one quiesce, PBS backup | +| **yes** | **yes** | **ONE quiesce window, BOTH backups inside it** | +| no | no | no quiesce | -Values were redacted at the point of collection — the probe compared `fingerprint`/`token` and emitted -booleans plus a 12-char digest; neither value left the box. +## 3. What shipped -### On the verbatim `agent channel recovered` line the brief asked for: it does not exist, by design +- **`quiesce.TieredBackend`** (optional extension to `Backend`) + `BackupTier`, + `ErrTiersUnsupported`. A backend that does not implement it drives the pre-R-82 path unchanged. +- **`agentapi`**: `BackupTiers`, `BackupDueFor`, `StartBackupFor`, `BackupStatusFor`. + `targetQuery("")` yields an **empty** suffix, so an untargeted call hits the untargeted route + byte-for-byte. +- **`Loop.resolveDueTiers`** — the dedup rule in one place, tiers in agent order. +- **`quiesceAndPollTiers` + `pollTier`** — one marker, one stop, N sequential backups, one resume. -`Check` returns early on a healthy probe — `return nil // healthy first-obs / steady-up → no notify` -— and `NotifyRecovered` fires only on a `down→up` transition **within one process's lifetime**. The -repair required a restart, so the new process's first observation is healthy and therefore silent. -There is no recovery line to quote, and reporting one would mean fabricating it. +## 4. Two decisions worth stating plainly -The positive evidence instead: +**The app stays quiesced until the LAST tier snapshots.** Resuming after tier 1's snapshot would +leave the following tier capturing a **running** app — losing app-consistency on exactly the DR tier +we most want it on. The consequence is user-visible and documented: on the both-due night downtime is +*(first tier's full backup)* + *(last tier's snapshot)*, not one snapshot. Tiers therefore run +**fast-first** — vzdump holds a guest lock so they are necessarily sequential, and the agent +advertises primary (local) first. The reverse order would make downtime ≈ *(offsite backup)* + +*(local snapshot)*, far worse. -- **zero** `[channel]` lines 90 s after restart on both boxes, where the pre-repair container logged - `transient down` within 60 s and confirmed `DOWN` within 120 s; -- the dashboard banner cleared (fetched through the authed endpoint, ASCII-safe grep); -- **zero** new `agent_channel_*` hub events since 06:45 UTC; -- demo-felhom health `ok` continuously. +**A manual "Mentés most" covers EVERY tier**, in one window, due-ness ignored. A manual run that +silently skipped the DR tier would be the same applied-and-empty fault in a different costume. -This is now written into the runbook, because "wait for the recovered line" would leave an operator -watching an empty log forever. +## 5. Capability detection ---- +`GET /backup/tiers` 404 ⇒ pre-R-82 agent. This is the project's documented **route-probe** mechanism +(`internal/agentapi/features.go`). It is deliberately **not** a `featureProbes` row: that table +answers a yes/no at a UI entry point, whereas the loop needs the tier **list**, so a row would be a +second probe of the same route for no gain. The degrade logs **exactly once per process** — once +because it is a steady state during a rollout, never zero times because a silent degrade is +indistinguishable from multi-tier working. -## 3. Files changed +## 6. v0.175.0 — a tier that overruns the quiesce bound defers the rest -**felhom-controller** (`9056f01`, plus `REPORT.md`): +Operator ruling 2026-07-26: *"let the first backup run as long as needed; other backups shouldn't +start until finished."* -| file | change | -|---|---| -| `internal/bootstrap/bootstrap.go` | `DetectEndpointDrift`, `EndpointDrift` + its two message builders | -| `internal/bootstrap/drift_test.go` | NEW — Scenarios A/B/C/D + the secret-leak assertions | -| `internal/web/alerts.go` | `endpointDriftAlert` field, `SetEndpointDriftAlert`, ordered ABOVE the channel banner in `GetAlerts` | -| `internal/notify/notifier.go` | `NotifyEndpointDrift` + `EndpointDriftDetails` | -| `cmd/controller/main.go` | startup wiring (once, after config settles) | -| `internal/monitor/healthcheck.go` | `EffectiveProtected` gate + the corrected doc comment | -| `internal/monitor/effective_protected_test.go` | Scenario E; existing toggle test updated | -| `internal/channelhealth/checker.go` | `stateUnconfirmed`, `orUnseeded`, re-arm condition, field comment | -| `internal/channelhealth/checker_test.go` | Scenario F ×2 | +A first FULL offsite snapshot legitimately runs for **hours**. When `max_quiesce` elapses the app +resumes — correct, unchanged — but the loop then started the **next** tier while the first was still +uploading. That is now a `break`. -**felhom.eu** (`9cfa619`): `hub/internal/api/handler.go` (allowlist), `hub/CHANGELOG.md`, -`manifests/hub.yaml`, `documentation/runbooks/RUNBOOK-local-api-endpoint-drift.md` (NEW), -ROADMAP (R-77 shipped + R-78/79/80), capability-map note. +Why it matters: vzdump still holds the guest lock, so the second start would be **refused by the +agent (409, v0.99.0)** or fail on the lock — and a failed backup never satisfies a cadence, so the +tier would stay permanently due and retry into the same wall every poll. -### Scope addition, flagged: the hub was NOT in the brief's scope, and had to be +## 7. Tests -`handleEvent` validates `event_type` against `allowedEventTypes` and returns -`400 Invalid event_type` otherwise. Shipping `local_api_endpoint_drift` controller-side alone would -have produced an alert that never reaches the operator — the inert-seam class this project has hit -four times, and precisely the failure mode R-77 exists to prevent. I added the one-line allowlist -entry, bumped the hub to v0.74.0 and deployed it. Live leg 2 proves the end-to-end path. +`go build ./... && go vet ./... && go test ./...` — **rc=0, 27 packages** (vet run unpiped; note +`grep FAIL` returns rc=1 when it finds nothing, which is not a failure signal — that idiom caused a +red commit elsewhere in this arc). ---- ++12 tests in `internal/quiesce/tiers_test.go`. Red-proofs observed and restored: -## 4. Tests + the three mandatory red-proofs +- **Both-due night** — a per-tier cycle instead of one window fails with + `want EXACTLY 1 stop and 1 start, got stops=2 starts=2`. **The COUNT is the assertion**; asserting + only "both backups ran" would pass against a double-quiesce implementation. +- **New controller ↔ old agent** — treating `ErrTiersUnsupported` as "nothing due" fails with + `OLD AGENT: a backup MUST still be taken via the untargeted path; got started=[]`. The hollow + version of this test asserts only "no error", which passes while silently skipping the backup. +- **Overrun defer** — dropping the `break` fails with + `the second tier MUST NOT start while the first is still running; started=[local felhom-pbs]`. -**951 → 959 test functions; 27 controller packages green; hub green.** -`go build ./... && go vet ./... && go test ./...` clean in both repos. +## 8. NOT live-validated — and NOT deployed -### Red-proof A — run in BOTH failure directions +The controller change is **unit-proven only**. Both production controllers still run **v0.173.0**; +this release was deliberately not deployed, because the multi-tier path only engages against an agent +that serves `/backup/tiers`, and the agent rollout was still mid-flight (demo-hp's first offsite +backup in progress). Specifically unproven live: -*(i) the pre-fix shape (v0.172.0: no detection at all):* -``` ---- FAIL: TestScenarioA_DriftDetectedAndNamed_NoWrite (0.00s) - drift_test.go:78: drift must be DETECTED — this is the exact live shape from the 2026-07-25 outage -``` -*(ii) the tempting wrong turn — a variant that detects **and helpfully corrects**, i.e. R-78 done -prematurely:* -``` ---- FAIL: TestScenarioA_DriftDetectedAndNamed_NoWrite (0.00s) - drift_test.go:101: controller.yaml was MODIFIED — detection must never write (that is R-78) -``` -The second is the one that matters: "an error was logged" is a hollow assertion; "and nothing was -written" is the contract. +1. A real both-due night with exactly one stop/start pair observed on a box. +2. The degrade path against a genuinely old agent in production. +3. The overrun defer firing on a real long-running offsite backup. -### Red-proof E -``` ---- FAIL: TestScenarioE_SambaProtectedOnlyWhenActuallyDeployed (0.00s) - effective_protected_test.go:99: (2) sharing ON, no password: samba protected = true, want false - — THE BUG: reconcileSambaAt refuses to deploy without a password — a deliberate state, not a fault -``` - -### Red-proof F -``` ---- FAIL: TestScenarioF_BornDownLogsUnseededNotUp (0.00s) - checker_test.go:308: an unseeded checker must not report state "up" after a suppressed first failure - checker_test.go:318: born-down must log "unseeded->down:unreachable"; got: - [WARN] [channel] agent channel DOWN (up->down:unreachable): dial tcp 192.168.0.87:8443: … -``` -And the non-change half: all **nine** pre-existing `channelhealth` tests still pass, including -`TestF2_BornDownNonTransient_AlertsOnce`, `TestF2_OldSeedSilentLogicWouldNotAlert` and -`TestF2_BornDownTransient_Debounced`. Scenario F asserts sink call **count and arguments** -(`reason`, `eventType`, `severity`, dashboard flag, probe count), not just the log string. - -### One pre-existing test was deliberately changed - -`TestEffectiveProtectedTracksSharingToggle` used `SMBSettings{Enabled: true}` and asserted samba WAS -watched — i.e. it encoded the bug. Updated to `Enabled: true, UserSet: true` with a comment saying -why, and the three-state matrix added alongside. Flagged here because "changed a passing test" always -deserves to be visible. - -### Gates - -`template_id_gate`, `emoji_gate`, `mojibake_gate`, `native_confirm_gate`, `app_row_dedup_gate`, -`offbox_rename_gate` — rc=0. `docker_run_volume_path_gate` **rc=1, PRE-EXISTING** (R-29(a), red since -v0.129.0, in `appexport/estimate.go` which this change does not touch). - ---- - -## 5. Deployed versions - -``` -demo-felhom 9201 : gitea.dooplex.hu/admin/felhom-controller:0.173.0 Up (healthy) -demo-hp 9201 : gitea.dooplex.hu/admin/felhom-controller:0.173.0 Up (healthy) -drill-r50 9201 : gitea.dooplex.hu/admin/felhom-controller:0.173.0 Up (healthy) -hub : gitea.dooplex.hu/admin/felhom-hub:0.74.0 Synced / Healthy, rolled out -``` - ---- - -## 6. Live legs - -**Leg 1 — no drift alert on the aligned production boxes (Scenario B live).** Zero `DRIFT` lines and -zero `[channel]` lines on both after the v0.173.0 restart. Agreement really is silent; the check does -not cry wolf on every healthy boot. - -**Leg 2 — `drill-r50`, the untouched real drift (Scenario A live).** It was left broken on purpose. - -Before (controller 0.161.0): -``` -controller.yaml sha256 BEFORE : 54892f1bc691c29b70ff9c323aab9e0b6ad758c602c533130064006e9a2af737 - controller.yaml endpoint : 192.168.0.176:8443 - bootstrap.json endpoint : 169.254.253.1:8443 - DRIFTED : True -``` -After deploying v0.173.0: -``` -[ERROR] bootstrap: local_api endpoint DRIFT — /opt/docker/felhom-controller/controller.yaml says -"192.168.0.176:8443" but /etc/felhom-bootstrap/bootstrap.json says "169.254.253.1:8443"; the -controller is dialling the FORMER. Pin agrees: true. Not auto-corrected (R-78 owns the authority -ruling) — fix the intended file and restart the controller. -[INFO] Event pushed: local_api_endpoint_drift (error) — … - -controller.yaml sha256 AFTER : 54892f1bc691c29b70ff9c323aab9e0b6ad758c602c533130064006e9a2af737 -``` -**Checksums identical — zero write, proven on a live divergence.** And hub-side, proving the -allowlist end-to-end: -``` -events: 2026-07-26 07:21:31 drill-r50 local_api_endpoint_drift error -notification_log: 2026-07-26 07:21:32 drill-r50 local_api_endpoint_drift operator sent -``` -`drill-r50` was **left drifted** — repairing it is the operator's call, and it is now the only live -fixture for this alert. - -**Leg 3 — demo-hp's false samba alarm is gone, sharing still on without a password.** -``` -[DEBUG] [monitor] Checking 4 protected containers: [traefik cloudflared felhom-controller filebrowser] -[DEBUG] [monitor] All protected containers running -[DEBUG] [monitor] Final status: ok (issues=0, warnings=0, info=5) -``` -Hub-side the flip is unambiguous: `fail` on 0.172.0 at 07:15:56 → **`ok` on 0.173.0 at 07:18:27**. - -**Leg 4 — the other half of Scenario E, live (operator-present state change).** Household password -set through the real `POST /sharing/password` endpoint (303): -``` -felhom-samba Up 45 seconds -[DEBUG] [monitor] Checking 5 protected containers: [traefik cloudflared felhom-controller filebrowser felhom-samba] -[DEBUG] [monitor] Final status: ok (issues=0, warnings=0, info=5) -``` -Samba deployed and **is watched again** — the fix suppresses the deliberate state, not the real one. - ---- - -## 7. NOT yet live-validated - -- **A drift where the FINGERPRINT also moved.** Unit-tested - (`TestDrift_FingerprintDisagreementIsSurfacedNotLeaked`); no live fixture exists, and manufacturing - one would mean deliberately corrupting a pin. -- **The drift banner rendered in a browser.** The alert and event are proven; the Hungarian dashboard - line is unit/`AlertManager`-level only. No browser on DooPlex. -- **Drift on a box where `controller.yaml` is the CORRECT file.** The detector is symmetric by - construction, but every live case so far has `bootstrap.json` correct. This is exactly the asymmetry - R-78 must rule on. -- **A recovery-line transition** (`down→up` within one process). The repair path restarts, so it is - structurally unreachable; the `recovered` branch is unit-tested only. -- **The samba gate through a full enable→password→disable cycle.** Enable-without-password and - password-set were both exercised; disabling sharing again was not. -- **Whether `drill-r50`'s channel actually recovers** once its endpoint is aligned — it was left - drifted on purpose. - ---- - -## 8. Observations — noticed, not acted on - -1. **`manifests/hub.yaml` was pinned to `0.73.1` while `hub/CHANGELOG.md` topped out at `0.73.2`** — - the documented "live image can lag the CHANGELOG" trap, found while bumping. v0.73.2's content is - in the 0.74.0 image now, but **v0.73.2 as a released artifact was never deployed**. Worth a glance - at what else it contained. -2. **`expected_backup_missed` still fires nightly on all three customers** — filed as **R-80**, with - the arithmetic the DIAG hedged on: 7.3 days of stale backup materially exceeds the ~1.5-day channel - outage, so backups were already stale ~6 days *before* the channel broke. R-77's repair will not - resolve it, and it reaches a **customer** channel. It plausibly outranks this task. -3. **`report.Issues`/`Warnings` remain English on a Hungarian surface** — R-79, whole-surface, - deliberately not swept here. -4. **The drift check is startup-only.** Correct today (both files are read at boot and neither changes - under a running controller), but if anything ever rewrites `bootstrap.json` live, the check would - not notice until the next restart. Named in the code comment so the assumption is visible. -5. **`mergeLocalAPI` replaces the whole block**, so endpoint/fingerprint/token cannot be reconciled - independently today. That constrains R-78's design space and is recorded in its ROADMAP entry. -6. **Sharing on demo-hp now has a household password and samba is running** (leg 4). That is a real, - deliberate state change from the brief, and it changes the box's LAN exposure — SMB is now actually - served. Flagging so it is a decision, not a surprise. +Deploying is the natural next step once demo-hp's first offsite backup lands and the agent rollout +is settled.