docs: REPORT/README/CONTEXT for v0.94.0 config-refresh (live-validated)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
This commit is contained in:
+14
-1
@@ -7,7 +7,20 @@
|
||||
>
|
||||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||||
|
||||
Last updated: 2026-06-29 (self-health arc COMPLETE — hub-side leaf-fp re-key detection)
|
||||
Last updated: 2026-06-30 (v0.94.0 — pull-based config-refresh; hub v0.26.0 retires the inbound controls)
|
||||
|
||||
> **2026-06-30 — v0.94.0 (deployed on 9201): pull-based config-refresh.** The hub report ACK now also
|
||||
> carries a per-customer **`config_version`** (hub v0.26.0; a stored counter bumped on every config save).
|
||||
> `OnPushResponse` → `ConfigRefresher.Reconcile` (`internal/report/config_refresh.go`): on a change vs.
|
||||
> `settings.applied_config_version`, `bootstrap.RefreshConfig` re-pulls `controller.yaml` (re-merging
|
||||
> `local_api` from bootstrap.json; overwrites controller.yaml, never settings.json) → record → graceful
|
||||
> self-restart (`api.GracefulSelfRestart`). First-run records baseline (no restart); unchanged = no-op
|
||||
> (no storm); failed pull keeps config + retries. This is the box-pulls-config replacement for the hub's
|
||||
> retired inbound "Push Config" — the hub never connects into the box. **Companion hub v0.26.0** also
|
||||
> retired Trigger Update / Pull Config / Show Diff and dropped geo-disable's inbound notify (kept the
|
||||
> hub→Cloudflare WAF removal), and fixed the stale `docker-setup.sh` setup command → host-install.
|
||||
> Live-validated on 9201: first-run baseline (applied=1, no restart) → DB bump 1→2 → re-pull+self-restart
|
||||
> (session_secret rotated, local_api preserved, RestartCount 0→1) → 4 cycles no-loop → apps stayed up.
|
||||
|
||||
> **2026-06-29 — Self-health arc complete (agent v0.48.0 + hub v0.22.1).** The hub now proactively
|
||||
> watches every agent's served local-API **leaf fingerprint** fleet-wide: the agent reports
|
||||
|
||||
@@ -1,84 +1,60 @@
|
||||
# REPORT — NAS Part B: off-box backup target (restic-over-SFTP), v0.93.0
|
||||
# REPORT — pull-based config-refresh (controller v0.94.0)
|
||||
|
||||
**Date:** 2026-06-30
|
||||
**Task:** `TASK.md` — NAS Part B (off-box restic-SFTP backup). Controller-only. Closes the NAS arc.
|
||||
**Task:** retire the hub's inbound controls + add pull-based config delivery. Controller half (companion:
|
||||
hub v0.26.0). The controller re-pulls `controller.yaml` + self-restarts when the report ACK's
|
||||
`config_version` changes — the box-pulls-config replacement for the retired "Push Config".
|
||||
|
||||
## Baseline
|
||||
| | value |
|
||||
|---|---|
|
||||
| felhom-controller | v0.92.0 → **v0.93.0** |
|
||||
| Commits | `2a7dead` (impl) · `02820d6` (restic sftp.command 0.14 fix) · `5e06254` (injection guard) |
|
||||
| restic in image | **re-added** (`restic` + `openssh-client`); live `restic 0.14.0` in the container |
|
||||
| Live image | `felhom-controller:0.93.0` on guest 9201 (`Up (healthy)`) |
|
||||
## Baselines
|
||||
- felhom-controller `464b14f` (v0.93.0) → `419d3d0` (**v0.94.0**, live on guest 9201 / felhom-pve).
|
||||
- Companion hub v0.26.0 live on k3s (advertises `config_version` in the report ACK).
|
||||
|
||||
## What was built
|
||||
## What changed
|
||||
- `internal/report/pusher.go`: `PushResponse.ConfigVersion` (`json:"config_version"`).
|
||||
- `internal/report/config_refresh.go` (NEW): `ConfigRefresher.Reconcile` — testable reconcile with
|
||||
injected side effects. change → Refresh→Record→Restart; first-run records baseline (no restart);
|
||||
unchanged = no-op; failed pull keeps config (no record/restart, retries); record-before-restart.
|
||||
- `internal/bootstrap/bootstrap.go`: `RefreshConfig` — re-pull + re-write controller.yaml, re-merging
|
||||
local_api from bootstrap.json (reuses pullWithRetry/mergeLocalAPI/writeFileAtomic). Overwrites
|
||||
controller.yaml (hub = source of truth); never touches settings.json; fail-safe.
|
||||
- `internal/settings/settings.go`: `applied_config_version` + Get/Set.
|
||||
- `internal/api/selfrestart.go`: exported `GracefulSelfRestart` (reuse; the unexported one calls it).
|
||||
- `cmd/controller/main.go`: wired `ConfigRefresher` into `OnPushResponse` beside the floor reconcile.
|
||||
|
||||
The customer's NAS as an **off-box backup destination**: an encrypted restic repo over SFTP holding the
|
||||
app-data tier (each off-box app's recovery unit + DB dumps + volume tars) — the "1 off-site" leg of 3-2-1,
|
||||
distinct from the local cross-drive rsync copy and the agent's PBS whole-CT DR. No kernel mount; restic
|
||||
talks SFTP to the NAS directly; the NAS sees only ciphertext.
|
||||
## Tests & green gate
|
||||
- `go build ./... && go vet ./... && go test ./...` — **all green**.
|
||||
- `Reconcile`: change→refresh+record+restart; **same-version NO-OP red-proof**; baseline-no-restart;
|
||||
failed-pull no-record/no-restart; zero-version no-op; record-fail skips restart.
|
||||
- `RefreshConfig`: re-pull overwrites + re-merges local_api; failed pull leaves config unchanged;
|
||||
absent bootstrap errors without writing.
|
||||
- **Red-proof (no-restart-storm):** removing the `ackVersion == applied` guard makes
|
||||
`TestReconcile_SameVersion_NoOp` FAIL (`refreshes=1/restarts=1, want 0`) — proving the guard prevents
|
||||
a restart-every-report loop. Restored → green.
|
||||
|
||||
### Files changed
|
||||
- `Dockerfile` — re-added `restic` + `openssh-client` (dropped when cross-drive migrated restic→rsync).
|
||||
- `internal/backup/offbox.go` (NEW) — restic-SFTP backend + orchestration:
|
||||
- **Fail-fast (spike Q8):** every restic call carries `-o sftp.command="ssh … -oConnectTimeout=10 … -s
|
||||
sftp"` so a dead NAS errors in ~10 s, never a multi-minute hang; `-oStrictHostKeyChecking=yes
|
||||
-oUserKnownHostsFile=<pinned>` (no blind TOFU) + `-oBatchMode=yes`.
|
||||
- init-if-absent (idempotent), per-app `restic backup --tag`, `forget --keep-daily 7 --keep-weekly 4
|
||||
--keep-monthly 6 --prune`, single-flight + migration-guard, restic's **own exit code** checked, restore
|
||||
via `restic restore latest --tag <app> --target <scratch>` (non-destructive).
|
||||
- **Secrets** (SSH key + auto-gen repo password) are **0600 files** in the data dir — never logged, never
|
||||
in a non-0600/committed file; the repo is encrypted. They ride DR via the PBS whole-CT snapshot.
|
||||
- **Injection guard** (`ValidateOffboxTarget`): host/user/repo are charset-checked and must not start
|
||||
with `-` (ssh option-injection, e.g. `-oProxyCommand`); `OffboxConfigured` fails closed on an invalid
|
||||
target so a hostile value can never reach the ssh exec.
|
||||
- `internal/settings/settings.go` — `OffboxTarget` + per-app `AppBackupPrefs.Offbox` + helpers (no secrets).
|
||||
- `cmd/controller/main.go` — daily `offbox-backup` (04:15), failure → `backup_failed` operator alert.
|
||||
- `internal/web/offbox_handlers.go` + `backups.html` — configure target (host/port/user/repo + out-of-band
|
||||
SSH key + known_hosts), per-app toggles, run-now, restore, status.
|
||||
## Live validation (guest 9201 on felhom-pve, hub.felhom.eu)
|
||||
Deployed 0.94.0 via the golden/bootstrap mechanism; push_interval temporarily 1m for fast cycles
|
||||
(restored to 15m after).
|
||||
|
||||
## Tests — `go build ./... && go vet ./... && go test ./...` GREEN
|
||||
- **ConnectTimeout fail-fast companion (red-proof):** a fake SSH transport hangs to the ctx deadline
|
||||
WITHOUT the arg, fails fast WITH it — proving the arg is load-bearing.
|
||||
- dead-NAS run fails fast + alerts + records status=error; restore round-trip byte-identical (SFTP-shaped
|
||||
seam); single-flight skip; repo init idempotency; secrets are 0600.
|
||||
- **Security companion:** `ValidateOffboxTarget` rejects ssh-option-injection (`-oProxyCommand` host/user),
|
||||
metacharacters, traversal, spaces, non-absolute paths; accepts a clean target.
|
||||
1. **First-run baseline (no restart):** first report after deploy recorded `applied_config_version=1`
|
||||
with `RestartCount=0` — baseline recorded, no restart.
|
||||
2. **Config edit converges:** bumped demo-felhom `config_version` 1→2 in the hub DB (equivalent to a
|
||||
config save). Next report ACK carried 2; controller logged:
|
||||
`config-refresh: hub config_version=2 != applied=1 — re-pulling controller.yaml` →
|
||||
`re-pulled … merged local_api (192.168.0.162:8443)` →
|
||||
`applied config_version=2 — self-restarting` → `Graceful self-restart: exiting (0)`.
|
||||
Container restarted (`RestartCount 0→1`, healthy); `applied_config_version=2`; the re-pull is proven
|
||||
by `web.session_secret` changing (`e1ed2dd3…` → `f456197e…`, regenerated by the hub each pull);
|
||||
`local_api` block preserved.
|
||||
3. **No restart storm:** 4 further report cycles at config_version=2 → zero config-refresh activity,
|
||||
container stayed up (no further restart).
|
||||
4. **Apps unaffected:** during the config-apply restart only `felhom-controller` restarted — every
|
||||
customer app (nextcloud, gitea, calcom, rallly, sparkyfitness, …) stayed `Up (healthy)`.
|
||||
5. **No inbound:** the whole flow ran with the hub never connecting into the box (it only answers the
|
||||
box's report); hub `grep client.Do internal/web/` is clean of ControllerURL targets.
|
||||
|
||||
## Live e2e validation (isolated sim NAS, SFTP)
|
||||
|
||||
**Method:** isolated throwaway sim = a Debian LXC on felhom-pve running sshd + a `felhom` user + `/srv/repo`
|
||||
(its own throwaway ed25519 key; never touched production storage). Drove the **real controller endpoints**
|
||||
(demo box has no password → CSRF skipped — the full server pipeline runs) via curl from inside the
|
||||
controller container; the off-box SSH key + known_hosts were injected through the config endpoint exactly as
|
||||
the UI form does. Sim + key destroyed afterward.
|
||||
|
||||
- **A — configure + first backup:** config saved (secrets written **0600**); off-box run **initialized the
|
||||
restic repo on the NAS** and backed up `rallly` (1 snapshot, 5 s). The NAS holds a proper restic repo
|
||||
(config/data/index/keys/snapshots); the config blob is **ciphertext** and a grep for SQL/dump/`rallly`
|
||||
markers in the repo found **nothing** — the NAS sees only ciphertext.
|
||||
- **B — dead NAS fails fast + alerts:** black-holed the sim (blackhole route in guest 9201, since the
|
||||
controller's egress is guest-side — the host iptables path used in A1/A2 doesn't catch guest→guest
|
||||
bridged traffic) → the run **failed in ~20 s** (= 2×ConnectTimeout for the cat-config + init probes) with
|
||||
restic's `connect to host … port 22: Connection timed out`; the **`backup_failed` operator alert fired**;
|
||||
the controller + guest stayed healthy (no hang). On route removal → a run succeeded again.
|
||||
- **C — restore byte-identical:** restored `rallly` from the off-box repo to a scratch dir →
|
||||
`diff -r` against the local source = **BYTE_IDENTICAL** (non-destructive; live data untouched).
|
||||
- **D — DR reachability:** the SSH key + repo password are **0600 files** in the controller data dir →
|
||||
captured by the agent's PBS whole-CT snapshot of the rootfs → a rebuilt box restores them and can
|
||||
list/restore the off-box repo. (The recovery-unit/dr-recipe stay secret-free by design; controller
|
||||
secrets ride DR via PBS — the existing escrow path.)
|
||||
- **Caught live (version skew):** restic 0.14.0 rejects `-o sftp.args` (the spike's form) — fixed to the
|
||||
portable `-o sftp.command="ssh … -s sftp"` (ConnectTimeout preserved), rebuilt, re-validated.
|
||||
- **Security finding (background commit review):** the injection guard above was added in response — then
|
||||
rebuilt + a regression backup confirmed a valid target still runs.
|
||||
|
||||
## Observations / follow-ups
|
||||
- **NAS arc complete** (A1 agent mount ✅ · A2 controller media registry ✅ · B off-box backup ✅).
|
||||
- Next queued: per-storage worst-fill alerting (hub host_disk follow-on) + a provisioning-side vzdump
|
||||
retention default so a box can't refill its own root.
|
||||
- DR-escrow note: off-box secrets ride the PBS whole-CT path (rootfs data dir), NOT the secret-free
|
||||
recovery-unit/dr-recipe — documented, no new escrow code.
|
||||
- The demo box's off-box target was disabled + its secrets removed at teardown (the sim is gone).
|
||||
|
||||
No secrets in any committed file. SSH key + repo password were throwaway and out-of-band only.
|
||||
## Notes
|
||||
- A config change rotates `web.session_secret` (configgen regenerates it per pull) → controller
|
||||
dashboard sessions are invalidated on a config apply. This is identical to the retired Push Config
|
||||
behaviour (same configgen path) — no regression; only on actual config changes, not every report.
|
||||
- The first-boot `bootstrap.MaybeIngest` never-clobber is untouched; the refresh is a separate explicit
|
||||
re-pull keyed on a version change.
|
||||
|
||||
@@ -1159,6 +1159,26 @@ The floor is the **auto-target** (the operator raises it for a controlled fleet
|
||||
the customer's manual opt-in. Floor source + operator UI are hub-side (felhom-hub v0.15.0). **No agent
|
||||
change — Phase 2 reuses the Phase 1 `POST /controller/swap`.**
|
||||
|
||||
##### Pull-based config-refresh (v0.94.0)
|
||||
|
||||
The same **report ACK** also carries a per-customer **`config_version`** (a hub-side stored counter,
|
||||
bumped on every config save; felhom-hub v0.26.0). This is how an operator config edit reaches a running
|
||||
box — **the hub never connects into the box** (it replaced the retired inbound "Push Config"). Wired in
|
||||
`OnPushResponse` beside the floor reconcile (`internal/report/config_refresh.go`, `ConfigRefresher.Reconcile`):
|
||||
|
||||
- **Changed** vs. the last-applied version (`settings.applied_config_version`) → `bootstrap.RefreshConfig`
|
||||
re-pulls `controller.yaml` from the hub and rewrites it, **re-merging the per-guest `local_api`** from
|
||||
`bootstrap.json` (reuses the first-boot pull machinery; overwrites `controller.yaml` since the hub is its
|
||||
source of truth; **never touches `settings.json`**) → record the new version → **graceful self-restart**
|
||||
(`api.GracefulSelfRestart` → `os.Exit(0)` → Docker `restart: unless-stopped` re-reads the new config).
|
||||
- **First-ever** ACK (nothing recorded) → record the baseline **without restarting** (the first-boot pull
|
||||
already has the current config).
|
||||
- **Unchanged** version → no-op (so no restart storm — after a refresh `applied == ACK`).
|
||||
- **Failed pull** → keep the current config, do **not** record/restart, retry next report cycle.
|
||||
- Record-before-restart so the restarted process sees the version applied and doesn't loop. Only the
|
||||
`felhom-controller` container restarts; customer app stacks are untouched. (A config apply rotates
|
||||
`web.session_secret`, so dashboard sessions are invalidated — same as the old Push-Config path.)
|
||||
|
||||
##### Design Philosophy
|
||||
|
||||
- **No automatic rollback** — follows the Watchtower pattern (24k+ GitHub stars, no rollback). Docker's `restart: unless-stopped` policy is the crash safety net. The Hub's dead man's switch detects when the controller goes down.
|
||||
|
||||
Reference in New Issue
Block a user