diff --git a/CONTEXT.md b/CONTEXT.md index 3edcdb2..4b26639 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,20 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-06-29 (self-health arc COMPLETE — hub-side leaf-fp re-key detection) +Last updated: 2026-06-30 (v0.94.0 — pull-based config-refresh; hub v0.26.0 retires the inbound controls) + +> **2026-06-30 — v0.94.0 (deployed on 9201): pull-based config-refresh.** The hub report ACK now also +> carries a per-customer **`config_version`** (hub v0.26.0; a stored counter bumped on every config save). +> `OnPushResponse` → `ConfigRefresher.Reconcile` (`internal/report/config_refresh.go`): on a change vs. +> `settings.applied_config_version`, `bootstrap.RefreshConfig` re-pulls `controller.yaml` (re-merging +> `local_api` from bootstrap.json; overwrites controller.yaml, never settings.json) → record → graceful +> self-restart (`api.GracefulSelfRestart`). First-run records baseline (no restart); unchanged = no-op +> (no storm); failed pull keeps config + retries. This is the box-pulls-config replacement for the hub's +> retired inbound "Push Config" — the hub never connects into the box. **Companion hub v0.26.0** also +> retired Trigger Update / Pull Config / Show Diff and dropped geo-disable's inbound notify (kept the +> hub→Cloudflare WAF removal), and fixed the stale `docker-setup.sh` setup command → host-install. +> Live-validated on 9201: first-run baseline (applied=1, no restart) → DB bump 1→2 → re-pull+self-restart +> (session_secret rotated, local_api preserved, RestartCount 0→1) → 4 cycles no-loop → apps stayed up. > **2026-06-29 — Self-health arc complete (agent v0.48.0 + hub v0.22.1).** The hub now proactively > watches every agent's served local-API **leaf fingerprint** fleet-wide: the agent reports diff --git a/REPORT.md b/REPORT.md index 590fbed..6733a70 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,84 +1,60 @@ -# REPORT — NAS Part B: off-box backup target (restic-over-SFTP), v0.93.0 +# REPORT — pull-based config-refresh (controller v0.94.0) **Date:** 2026-06-30 -**Task:** `TASK.md` — NAS Part B (off-box restic-SFTP backup). Controller-only. Closes the NAS arc. +**Task:** retire the hub's inbound controls + add pull-based config delivery. Controller half (companion: +hub v0.26.0). The controller re-pulls `controller.yaml` + self-restarts when the report ACK's +`config_version` changes — the box-pulls-config replacement for the retired "Push Config". -## Baseline -| | value | -|---|---| -| felhom-controller | v0.92.0 → **v0.93.0** | -| Commits | `2a7dead` (impl) · `02820d6` (restic sftp.command 0.14 fix) · `5e06254` (injection guard) | -| restic in image | **re-added** (`restic` + `openssh-client`); live `restic 0.14.0` in the container | -| Live image | `felhom-controller:0.93.0` on guest 9201 (`Up (healthy)`) | +## Baselines +- felhom-controller `464b14f` (v0.93.0) → `419d3d0` (**v0.94.0**, live on guest 9201 / felhom-pve). +- Companion hub v0.26.0 live on k3s (advertises `config_version` in the report ACK). -## What was built +## What changed +- `internal/report/pusher.go`: `PushResponse.ConfigVersion` (`json:"config_version"`). +- `internal/report/config_refresh.go` (NEW): `ConfigRefresher.Reconcile` — testable reconcile with + injected side effects. change → Refresh→Record→Restart; first-run records baseline (no restart); + unchanged = no-op; failed pull keeps config (no record/restart, retries); record-before-restart. +- `internal/bootstrap/bootstrap.go`: `RefreshConfig` — re-pull + re-write controller.yaml, re-merging + local_api from bootstrap.json (reuses pullWithRetry/mergeLocalAPI/writeFileAtomic). Overwrites + controller.yaml (hub = source of truth); never touches settings.json; fail-safe. +- `internal/settings/settings.go`: `applied_config_version` + Get/Set. +- `internal/api/selfrestart.go`: exported `GracefulSelfRestart` (reuse; the unexported one calls it). +- `cmd/controller/main.go`: wired `ConfigRefresher` into `OnPushResponse` beside the floor reconcile. -The customer's NAS as an **off-box backup destination**: an encrypted restic repo over SFTP holding the -app-data tier (each off-box app's recovery unit + DB dumps + volume tars) — the "1 off-site" leg of 3-2-1, -distinct from the local cross-drive rsync copy and the agent's PBS whole-CT DR. No kernel mount; restic -talks SFTP to the NAS directly; the NAS sees only ciphertext. +## Tests & green gate +- `go build ./... && go vet ./... && go test ./...` — **all green**. +- `Reconcile`: change→refresh+record+restart; **same-version NO-OP red-proof**; baseline-no-restart; + failed-pull no-record/no-restart; zero-version no-op; record-fail skips restart. +- `RefreshConfig`: re-pull overwrites + re-merges local_api; failed pull leaves config unchanged; + absent bootstrap errors without writing. +- **Red-proof (no-restart-storm):** removing the `ackVersion == applied` guard makes + `TestReconcile_SameVersion_NoOp` FAIL (`refreshes=1/restarts=1, want 0`) — proving the guard prevents + a restart-every-report loop. Restored → green. -### Files changed -- `Dockerfile` — re-added `restic` + `openssh-client` (dropped when cross-drive migrated restic→rsync). -- `internal/backup/offbox.go` (NEW) — restic-SFTP backend + orchestration: - - **Fail-fast (spike Q8):** every restic call carries `-o sftp.command="ssh … -oConnectTimeout=10 … -s - sftp"` so a dead NAS errors in ~10 s, never a multi-minute hang; `-oStrictHostKeyChecking=yes - -oUserKnownHostsFile=` (no blind TOFU) + `-oBatchMode=yes`. - - init-if-absent (idempotent), per-app `restic backup --tag`, `forget --keep-daily 7 --keep-weekly 4 - --keep-monthly 6 --prune`, single-flight + migration-guard, restic's **own exit code** checked, restore - via `restic restore latest --tag --target ` (non-destructive). - - **Secrets** (SSH key + auto-gen repo password) are **0600 files** in the data dir — never logged, never - in a non-0600/committed file; the repo is encrypted. They ride DR via the PBS whole-CT snapshot. - - **Injection guard** (`ValidateOffboxTarget`): host/user/repo are charset-checked and must not start - with `-` (ssh option-injection, e.g. `-oProxyCommand`); `OffboxConfigured` fails closed on an invalid - target so a hostile value can never reach the ssh exec. -- `internal/settings/settings.go` — `OffboxTarget` + per-app `AppBackupPrefs.Offbox` + helpers (no secrets). -- `cmd/controller/main.go` — daily `offbox-backup` (04:15), failure → `backup_failed` operator alert. -- `internal/web/offbox_handlers.go` + `backups.html` — configure target (host/port/user/repo + out-of-band - SSH key + known_hosts), per-app toggles, run-now, restore, status. +## Live validation (guest 9201 on felhom-pve, hub.felhom.eu) +Deployed 0.94.0 via the golden/bootstrap mechanism; push_interval temporarily 1m for fast cycles +(restored to 15m after). -## Tests — `go build ./... && go vet ./... && go test ./...` GREEN -- **ConnectTimeout fail-fast companion (red-proof):** a fake SSH transport hangs to the ctx deadline - WITHOUT the arg, fails fast WITH it — proving the arg is load-bearing. -- dead-NAS run fails fast + alerts + records status=error; restore round-trip byte-identical (SFTP-shaped - seam); single-flight skip; repo init idempotency; secrets are 0600. -- **Security companion:** `ValidateOffboxTarget` rejects ssh-option-injection (`-oProxyCommand` host/user), - metacharacters, traversal, spaces, non-absolute paths; accepts a clean target. +1. **First-run baseline (no restart):** first report after deploy recorded `applied_config_version=1` + with `RestartCount=0` — baseline recorded, no restart. +2. **Config edit converges:** bumped demo-felhom `config_version` 1→2 in the hub DB (equivalent to a + config save). Next report ACK carried 2; controller logged: + `config-refresh: hub config_version=2 != applied=1 — re-pulling controller.yaml` → + `re-pulled … merged local_api (192.168.0.162:8443)` → + `applied config_version=2 — self-restarting` → `Graceful self-restart: exiting (0)`. + Container restarted (`RestartCount 0→1`, healthy); `applied_config_version=2`; the re-pull is proven + by `web.session_secret` changing (`e1ed2dd3…` → `f456197e…`, regenerated by the hub each pull); + `local_api` block preserved. +3. **No restart storm:** 4 further report cycles at config_version=2 → zero config-refresh activity, + container stayed up (no further restart). +4. **Apps unaffected:** during the config-apply restart only `felhom-controller` restarted — every + customer app (nextcloud, gitea, calcom, rallly, sparkyfitness, …) stayed `Up (healthy)`. +5. **No inbound:** the whole flow ran with the hub never connecting into the box (it only answers the + box's report); hub `grep client.Do internal/web/` is clean of ControllerURL targets. -## Live e2e validation (isolated sim NAS, SFTP) - -**Method:** isolated throwaway sim = a Debian LXC on felhom-pve running sshd + a `felhom` user + `/srv/repo` -(its own throwaway ed25519 key; never touched production storage). Drove the **real controller endpoints** -(demo box has no password → CSRF skipped — the full server pipeline runs) via curl from inside the -controller container; the off-box SSH key + known_hosts were injected through the config endpoint exactly as -the UI form does. Sim + key destroyed afterward. - -- **A — configure + first backup:** config saved (secrets written **0600**); off-box run **initialized the - restic repo on the NAS** and backed up `rallly` (1 snapshot, 5 s). The NAS holds a proper restic repo - (config/data/index/keys/snapshots); the config blob is **ciphertext** and a grep for SQL/dump/`rallly` - markers in the repo found **nothing** — the NAS sees only ciphertext. -- **B — dead NAS fails fast + alerts:** black-holed the sim (blackhole route in guest 9201, since the - controller's egress is guest-side — the host iptables path used in A1/A2 doesn't catch guest→guest - bridged traffic) → the run **failed in ~20 s** (= 2×ConnectTimeout for the cat-config + init probes) with - restic's `connect to host … port 22: Connection timed out`; the **`backup_failed` operator alert fired**; - the controller + guest stayed healthy (no hang). On route removal → a run succeeded again. -- **C — restore byte-identical:** restored `rallly` from the off-box repo to a scratch dir → - `diff -r` against the local source = **BYTE_IDENTICAL** (non-destructive; live data untouched). -- **D — DR reachability:** the SSH key + repo password are **0600 files** in the controller data dir → - captured by the agent's PBS whole-CT snapshot of the rootfs → a rebuilt box restores them and can - list/restore the off-box repo. (The recovery-unit/dr-recipe stay secret-free by design; controller - secrets ride DR via PBS — the existing escrow path.) -- **Caught live (version skew):** restic 0.14.0 rejects `-o sftp.args` (the spike's form) — fixed to the - portable `-o sftp.command="ssh … -s sftp"` (ConnectTimeout preserved), rebuilt, re-validated. -- **Security finding (background commit review):** the injection guard above was added in response — then - rebuilt + a regression backup confirmed a valid target still runs. - -## Observations / follow-ups -- **NAS arc complete** (A1 agent mount ✅ · A2 controller media registry ✅ · B off-box backup ✅). -- Next queued: per-storage worst-fill alerting (hub host_disk follow-on) + a provisioning-side vzdump - retention default so a box can't refill its own root. -- DR-escrow note: off-box secrets ride the PBS whole-CT path (rootfs data dir), NOT the secret-free - recovery-unit/dr-recipe — documented, no new escrow code. -- The demo box's off-box target was disabled + its secrets removed at teardown (the sim is gone). - -No secrets in any committed file. SSH key + repo password were throwaway and out-of-band only. +## Notes +- A config change rotates `web.session_secret` (configgen regenerates it per pull) → controller + dashboard sessions are invalidated on a config apply. This is identical to the retired Push Config + behaviour (same configgen path) — no regression; only on actual config changes, not every report. +- The first-boot `bootstrap.MaybeIngest` never-clobber is untouched; the refresh is a separate explicit + re-pull keyed on a version change. diff --git a/controller/README.md b/controller/README.md index 7bb2fa1..bc9510e 100644 --- a/controller/README.md +++ b/controller/README.md @@ -1159,6 +1159,26 @@ The floor is the **auto-target** (the operator raises it for a controlled fleet the customer's manual opt-in. Floor source + operator UI are hub-side (felhom-hub v0.15.0). **No agent change — Phase 2 reuses the Phase 1 `POST /controller/swap`.** +##### Pull-based config-refresh (v0.94.0) + +The same **report ACK** also carries a per-customer **`config_version`** (a hub-side stored counter, +bumped on every config save; felhom-hub v0.26.0). This is how an operator config edit reaches a running +box — **the hub never connects into the box** (it replaced the retired inbound "Push Config"). Wired in +`OnPushResponse` beside the floor reconcile (`internal/report/config_refresh.go`, `ConfigRefresher.Reconcile`): + +- **Changed** vs. the last-applied version (`settings.applied_config_version`) → `bootstrap.RefreshConfig` + re-pulls `controller.yaml` from the hub and rewrites it, **re-merging the per-guest `local_api`** from + `bootstrap.json` (reuses the first-boot pull machinery; overwrites `controller.yaml` since the hub is its + source of truth; **never touches `settings.json`**) → record the new version → **graceful self-restart** + (`api.GracefulSelfRestart` → `os.Exit(0)` → Docker `restart: unless-stopped` re-reads the new config). +- **First-ever** ACK (nothing recorded) → record the baseline **without restarting** (the first-boot pull + already has the current config). +- **Unchanged** version → no-op (so no restart storm — after a refresh `applied == ACK`). +- **Failed pull** → keep the current config, do **not** record/restart, retry next report cycle. +- Record-before-restart so the restarted process sees the version applied and doesn't loop. Only the + `felhom-controller` container restarts; customer app stacks are untouched. (A config apply rotates + `web.session_secret`, so dashboard sessions are invalidated — same as the old Push-Config path.) + ##### Design Philosophy - **No automatic rollback** — follows the Watchtower pattern (24k+ GitHub stars, no rollback). Docker's `restart: unless-stopped` policy is the crash safety net. The Hub's dead man's switch detects when the controller goes down.