admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: v0.75.0 REPORT/CONTEXT — userdata mountpoint gate + boot-ordering design note + live C6 proof

Browse Source 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-06-22 20:55:08 +02:00
parent 0b2605c5a8
commit 8b4b9011dd
2 changed files with 64 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+22
View File
@@ -7,6 +7,28 @@
> >
> Ask Claude Code: "Please update CONTEXT.md with what we did today" > Ask Claude Code: "Please update CONTEXT.md with what we did today"
Last updated: 2026-06-22 (v0.75.0 — gate userdata MkdirAll on a live mountpoint)
> **2026-06-22 — v0.75.0 (deployed on 9201): userdata MkdirAll gated on a live mountpoint.**
> Both `MkdirAll`-into-`<drive>/userdata` sites — the deploy belt (`stacks.ensureUserdataMounts`) and
> the FileBrowser sync (`web.syncFileBrowserMounts`) — now skip when an **external** drive root (under
> `StableParentDir=/mnt/felhom-drives`, `!= sysDataPath`) is **not a live mountpoint** (`system.IsMountPoint`).
> Closes the campaign-#2 hazard where a drive-absent window produced `mkdir …/userdata: permission denied`
> + transient `Created` flapping AND could write app data onto the guest **rootfs** (shadowed when the
> drive returns). The app is held by `planDriveGates` instead. System/local path is never gated. New
> `Manager.isMountPoint` seam + pure `web.skipFileBrowserPath` helper for tests. Live-proven via a C6
> re-run: 0 permission-denied, 0 shadow dirs on the rootfs, apps recover on reconnect.
>
> **Boot-ordering design note (NOT implemented — decide separately):** the *boot-time* `mkdir …
> permission denied` is a different cause — **docker's** boot-restore auto-starts drive-backed
> containers (`restart: unless-stopped`) before the agent mounts the drives, so docker (not the belt)
> tries to create the bind source; `planDriveGates` recovers them after mount convergence (why a reboot
> ends healthy). Options: (A) accept + suppress (apps self-heal; lowest risk; maybe downgrade the
> boot-window log level) — recommended now; (B) drive-backed app containers don't docker-auto-start at
> boot (restart policy `no`/`on-failure`) so the controller's gate is the sole starter after mounts
> converge — cleaner but bigger (must cover crash-restart too). See
> `felhom.eu/documentation/tests/test-campaign-2-finding1-recovery-diagnosis.md` (boot-ordering observation).
Last updated: 2026-06-22 (v0.74.0 — controller→agent connection-leak fix) Last updated: 2026-06-22 (v0.74.0 — controller→agent connection-leak fix)
> **2026-06-22 — v0.74.0 (deployed on demo guest 9201): fixed the controller→agent connection leak.** > **2026-06-22 — v0.74.0 (deployed on demo guest 9201): fixed the controller→agent connection leak.**
REPORT.md
+42 -51
View File
@@ -1,64 +1,55 @@
# REPORT — controller v0.74.0: fix the controller→agent connection leak # REPORT — controller v0.75.0: gate userdata MkdirAll on a live mountpoint
## Baseline → target ## Baseline → target
`felhom-controller` `main` v0.73.0 → **v0.74.0**. Controller-only change; agent/hub/firewall untouched. `felhom-controller` `main` v0.74.0 → **v0.75.0**. Controller-only; agent/hub untouched.
## Problem ## Problem
`Server.agentClient()` built a fresh `agentapi.Client` (hence a fresh bare `http.Transport`, Two `MkdirAll`-into-`<drive>/userdata` sites fired without checking the drive was mounted (campaign-#2
`IdleConnTimeout:0`) on **every** agent API call and discarded it without closing idle connections. findings #2/#3): the deploy belt (`ensureUserdataMounts`) and the FileBrowser sync. During a
The agent's keep-alive left one idle `ESTABLISHED` socket per call to `192.168.0.162:8443`; they drive-absent window they logged `mkdir …/userdata: permission denied` + caused transient `Created`
accumulated (~5.8k/day) until the ephemeral source-port range for that tuple exhausted → container flapping — and worse, writing into an unmounted mountpoint lands app data on the guest
`connect: cannot assign requested address` (EADDRNOTAVAIL), disabling storage UI, host-metrics, and **rootfs**, shadowed when the drive returns (data-integrity + rootfs-fill hazard).
whole-guest backup after ~5 days of controller uptime. (`:8006`/pveproxy was immune — the controller
never dials it.) Diagnosis:
`felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-8443-diagnosis.md`.
## Change (commit `2a5b88f`) ## Change (commit `0b2605c`)
- `internal/web/server.go` — `Server` gains `agentCli *agentapi.Client`, `agentCliErr error`, - `internal/stacks/manager.go` — `ensureUserdataMounts` skips when the `HDD_PATH` drive root is an
`agentCliOnce sync.Once` (+ the `agentapi` import). **external** path (`!= sysDataPath`) that is **not a live mountpoint**; the app is held by
- `internal/web/agent_disk_handlers.go` — `agentClient()` memoizes the build via `agentCliOnce` and `planDriveGates`. New injectable `Manager.isMountPoint` seam (defaults to `system.IsMountPoint`).
**returns one shared client** (cfg.LocalAPI is static per process; a config-apply self-restarts the The system/local path is never gated.
controller). The empty-endpoint "not configured" guard stays OUTSIDE the `Once`. Signature and all - `internal/web/handlers.go` — the FileBrowser sync loop skips (and doesn't mount) a path under
19 call sites unchanged; `*agentapi.Client`/`*http.Client` are concurrency-safe so no extra locking. `StableParentDir` that isn't a live mountpoint, via the new pure `skipFileBrowserPath` helper.
- `internal/agentapi/client.go` — `New` Transport hardened: `MaxIdleConns:4`, - `EnsureUserdataDir`/`EnsureUserdataSkeleton`/`planDriveGates` unchanged (gated the callers).
`MaxIdleConnsPerHost:2`, `IdleConnTimeout:90s` (was bare, `IdleConnTimeout:0`). Added optional
`Client.Close()` (CloseIdleConnections) hygiene helper.
## Tests (green; both red-proofed) ## Tests (green; red-proofed)
- `go build ./... && go vet ./... && go test ./...` — all green. `go build ./... && go vet ./... && go test ./...` all green.
- **T1** `TestAgentClient_ReusesSameInstance` (web) — two `agentClient()` calls return the identical - T1 `TestEnsureUserdataMounts_SkipsAbsentExternalDrive` — belt skips when external drive absent.
pointer; `TestAgentClient_UnconfiguredErrors` — empty endpoint still errors. Red-proof: removing the gate → dir created → FAIL (shown, reverted).
Red-proof: reverting to per-call `agentapi.New` → pointers differ → FAIL (shown, reverted). - T2 `TestEnsureUserdataMounts_EnsuresWhenMounted` — belt ensures when drive mounted.
- **T2** `TestNew_TransportIdlePoolBounded` (agentapi, white-box) — `IdleConnTimeout>0` AND - T3 `TestEnsureUserdataMounts_SystemPathNeverSkipped` — system/local path never gated (must-not-over-gate).
`MaxIdleConnsPerHost>0`. Red-proof: bare Transport → `IdleConnTimeout==0` → FAIL (shown, reverted). - T4 `TestSkipFileBrowserPath` — only an absent external path is skipped; mounted + system paths kept.
Red-proof: gate→false → absent `usb` not skipped → FAIL (shown, reverted).
## Deploy ## Deploy
Built+pushed `gitea.dooplex.hu/admin/felhom-controller:0.74.0` on 192.168.0.180 (digest Built+pushed `gitea.dooplex.hu/admin/felhom-controller:0.75.0` on 192.168.0.180 (digest
`sha256:2e85376e…`), deployed to guest 9201 via the bootstrap path (`docker pull` → pin `sha256:f735676e…`; **build-server `git pull` done first** — the v0.74.0 stale-source gotcha),
`/etc/felhom-controller-image` → restart `felhom-controller-bootstrap.service`). `docker inspect`: deployed to guest 9201 via the bootstrap path. `docker inspect`: `image=:0.75.0 running healthy`.
`image=:0.74.0 status=running health=healthy`.
> Process note: the first build packaged stale source (the build server's `~/git/felhom-controller`
> was at v0.73.0 — the required `git -C ~/git/felhom-controller pull` step had been skipped, so the
> image was the old code mislabeled `:0.74.0`; the live test still leaked). Pulled the source to
> `2a5b88f` and rebuilt — second image (`sha256:2e85376e…`) is the real fix.
## LIVE acceptance — leak is gone (the real proof) ## LIVE acceptance — campaign-#2 C6 re-run, now clean
Burst of **120 agent calls** (alternating `/api/disks` + `/api/host-metrics`) against the controller in Disconnected felhom-flash (8 apps) and observed the drive-absent window:
guest 9201, sampling idle `ESTABLISHED` sockets to `192.168.0.162:8443` from the controller's netns
(`nsenter -t <pid> -n ss -tn state established dst 192.168.0.162:8443 | grep -c 192.168`):
| stage | pre-fix image (stale build) | **fixed image (2e85376e)** | | Check | campaign #2 (pre-fix) | **v0.75.0 (fixed)** |
|---|---|---| |---|---|---|
| before | 8 | **1** | | `mkdir …/userdata: permission denied` | many | **0** |
| after 40 calls | 48 | **2** | | "could not ensure userdata skeleton" | present | **0** |
| after 80 calls | 90 | **2** | | rootfs shadow dir at unmounted mountpoint | created | **none** — `/mnt/felhom-drives/felhom-flash/userdata` does not exist (bare mountpoint) |
| after 120 calls | 132 | **2** | | apps during absent window | transient `Created` flapping | held cleanly (down) |
| reconnect | recovered | **recovered: flash remounted /dev/sdc1, 8 apps restarted → 25 running / 0 unhealthy** |
**Fixed: flat at 2 (= MaxIdleConnsPerHost), independent of call count.** Pre-fix grew ~1/call. Agent The integrity hazard (app data shadowed onto the rootfs) is closed: the now-unmounted mountpoint is
endpoints still return real data (`/api/disks` lists disks, `/api/host-metrics` returns host CPU/mem). empty, no `userdata/...` was written to the rootfs.
## Not touched / separate open item ## Not touched / design note (see CONTEXT.md)
The agent, its bridge-IP `ListenAddr` bind (deliberate defense-in-depth), and all firewall rules were The **boot-time** `mkdir … permission denied` (daemon boot-restore) is a different cause — **docker**
**not** changed (controller-only fix). **Separate open item (NOT addressed here):** the defense-in-depth auto-starts drive-backed containers (`restart: unless-stopped`) before the agent mounts the drives, so
host firewall rule scoping `:8443` to the guest bridge subnet is still absent (`pve-firewall` disabled, docker (not the belt) tries to create the bind source; `planDriveGates` recovers them after mount
no 8443 rule) — close it independently of this fix. convergence. This task ships only the runtime gate; the boot-ordering decision is documented, not
implemented.