204 lines
15 KiB
Markdown
204 lines
15 KiB
Markdown
# REPORT — Live demo validation of felhom-controller v0.39.0 + 8C orphan-template cleanup (2026-06-11)
|
|
|
|
Two things this session: (1) **provisioned a fresh customer guest from a v0.39.0 golden on the demo
|
|
Proxmox host and walked the full controller flow**, reporting what works vs breaks against the live
|
|
guest; (2) a small source-hygiene code change — deleting five dead 8C orphan templates (v0.39.1).
|
|
|
|
The code change is **source-only** (ships in the next golden); the running demo guest stays the
|
|
v0.39.0 golden provisioned below.
|
|
|
|
---
|
|
|
|
## Premise correction (surfaced to Viktor, decision taken)
|
|
|
|
The session brief said "controller v0.39.0 is built and baked into the current golden." That was
|
|
**false on the baked half**:
|
|
- v0.39.0 **source** committed (`d8d1e17`, slice 9). ✓
|
|
- v0.39.0 **image** built & pushed to the registry — but under tag **`0.39.0`** (no `v` prefix,
|
|
unlike every prior build; same digest as `latest`). The `v0.39.0` alias was **never pushed**.
|
|
- v0.39.0 **golden** did **not exist** — the newest golden baked **v0.38.0** (others v0.37.0/.36/.35).
|
|
|
|
Per the plan's Step-1.6 gate, this was surfaced. **Viktor chose: re-bake the v0.39.0 golden, then run
|
|
the full session.** Done (Step 1b below).
|
|
|
|
---
|
|
|
|
## Step 1 — Discovery (observed)
|
|
|
|
| Item | Value |
|
|
|---|---|
|
|
| Live agent config path | `/root/.config/felhom-agent/agent.json` (NOT the `/etc/...` default — confirmed from `systemctl cat felhom-agent` `ExecStart`) |
|
|
| Agent version (deployed) | **v0.18.0** (serves `GET /host/metrics`, `/disks`, `/disks/format`) |
|
|
| `local_api` | `enable: true`, `listen_addr: 192.168.0.162:8443` ✓ |
|
|
| `backup.restore_storage` | `local-lvm` ✓ |
|
|
| `hub` | `url: https://hub.felhom.eu`, `host_id: demo-felhom-01`, `api_key` set (stored out-of-band) ✓ |
|
|
| Free VMID chosen | **9200** (in use: 9001 spike, 9999 selftest-scratch) |
|
|
| Golden volid (re-baked) | `local:backup/vzdump-lxc-9100-2026_06_11-11_55_03.tar.zst` (baked image `gitea.dooplex.hu/admin/felhom-controller:0.39.0`, verified by extracting `/etc/felhom-controller-image` from the archive) |
|
|
| Hub row key | Controller report keys on `customer.id`; the existing **`demo-felhom` customer row exists** (stale, last reports Feb 2026 up to v0.28.8). Agent host-report keys on `host_id=demo-felhom-01` (separate row). |
|
|
| Daemon contention | **None at mint time** — provision ran with the daemon up, no token-store/leaf lock. (But see the post-provision 401 finding under Step 2.) |
|
|
|
|
### Step 1b — re-bake the v0.39.0 golden
|
|
- The supplied Gitea token (read-only) **lacks container-registry scope** — the build LXC's
|
|
`docker pull` failed `401 unauthorized` (verified: the token gets HTTP 401 from the registry token
|
|
endpoint). Re-used the **build server's existing registry credential** (package-scoped, the same one
|
|
prior bakes used; **stored out-of-band**) purely inside the build LXC. `build-golden.sh` logs out +
|
|
removes `/root/.docker/config.json` before archiving, so **no credential is baked** into the golden;
|
|
the build logfile contains no secret.
|
|
- New golden built (VMID 9100 build LXC → archived → build guest destroyed). Baked image confirmed
|
|
`:0.39.0`.
|
|
- **Follow-up for Viktor:** push the `v0.39.0` tag alias for naming convention, and bump the
|
|
`build-golden.sh:33` default off the stale `:v0.35.0`.
|
|
|
|
---
|
|
|
|
## Step 2 — Provision guest 9200 as `demo-felhom`
|
|
|
|
`felhom-agent --selftest=provision -config /root/.config/felhom-agent/agent.json -archive <golden>
|
|
-vmid 9200 -customer-id demo-felhom -customer-domain demo-felhom.eu -customer-name "Felhom Demo"
|
|
-hostname felhom-demo` → `selftest=provision OK — guest 9200 provisioned + bootstrap-mounted (KEPT)`.
|
|
|
|
**`pct reboot 9200` was mandatory** (the bootstrap oneshot is gated on the mount existing at boot, and
|
|
provision attaches the mount *after* the front-half boot). After one reboot the controller deployed:
|
|
|
|
| Assertion | Result |
|
|
|---|---|
|
|
| Controller running | `gitea.dooplex.hu/admin/felhom-controller:0.39.0 Up (healthy)` — startup line `felhom-controller 0.39.0 starting (customer: demo-felhom, domain: demo-felhom.eu)` |
|
|
| Not setup mode | ✓ — no `setup mode` line; came up **configured** |
|
|
| Bootstrap-seeded config | ✓ — `controller.yaml` written **0600 root:root** at boot, with customer id + hub + `local_api.{endpoint,fingerprint,token}` |
|
|
| De-privileged container | ✓ — `Privileged=false`; mounts = **exactly 3**: `/etc/felhom-bootstrap` (ro), `felhom-controller-data` volume (rw), `/var/run/docker.sock` (rw). No `/dev`, `/etc/fstab`, `/mnt` rshared, `/sys`, `/run/udev`. LXC `unprivileged=1`, features `nesting,keyctl` only, single `mp9` bootstrap mount, no device passthrough/hookscript. |
|
|
| No registry pull | ✓ — bootstrap unit did `docker run` (no pull); image shows `created 20 hours ago` (golden bake time) |
|
|
| bootstrap.json identity | ✓ — `/etc/felhom-bootstrap/bootstrap.json` `600 root:root`, `customer.id=demo-felhom`, hub creds, `local_api.{endpoint=192.168.0.162:8443, fingerprint=60b5974d…, token}` (token/api_key out-of-band) |
|
|
|
|
### FINDING 2a — local-API channel 401 until the agent daemon is restarted
|
|
At controller startup: `local-api: GET /storage failed (agentapi: GET /storage: HTTP 401) — channel
|
|
not verified`. Root cause: provision minted the per-guest token and durably recorded its hash
|
|
(`/var/lib/felhom-agent/local-tokens.log` carries `{"v":9200,...}`), but the **already-running daemon
|
|
loaded its in-memory token map at its own startup, before the mint** — so it rejected the controller's
|
|
token. **`systemctl restart felhom-agent` cleared it** (the daemon reloads the durable store on
|
|
restart; minted hash persists). After restart, all agent-channel calls succeed.
|
|
**Recommendation:** provision should signal the running daemon to reload the token store (or the
|
|
local-API should consult the durable store per-request), OR the provisioning runbook must include a
|
|
post-provision `systemctl restart felhom-agent`. As-is, a guest provisioned while the daemon runs has a
|
|
dead local-API channel until the daemon restarts.
|
|
|
|
### FINDING 2b — bootstrap seed-log line absent (cosmetic)
|
|
The seed functionally worked (controller.yaml 0600 written at boot, configured, not setup mode), but
|
|
the explicit `[INFO] bootstrap: seeded … coming up configured` line (`bootstrap/bootstrap.go:111`) did
|
|
**not** appear in `docker logs`. Functionally correct; logging-only discrepancy worth a glance.
|
|
|
|
---
|
|
|
|
## Step 3 — Full flow (validated from inside the container via `docker exec … curl localhost:8080`)
|
|
|
|
The bootstrap `docker run` publishes no port (bridge-only), so host-`localhost` is refused — all checks
|
|
ran inside the container (host=localhost passes the catch-all gate).
|
|
|
|
1. **UI renders** — all HTTP 200 with Hungarian markers: `/` **Vezérlőpult** (open dashboard — no
|
|
web password set, logged `no password configured — dashboard is open`), `/stacks` (Alkalmazások),
|
|
`/monitoring` **Rendszermonitor** + the slice-9 **"Szerver állapota (gazdagép)"** host card,
|
|
`/backups` (Biztonsági mentés), `/settings` (Beállítások).
|
|
- **Host-gate routing proven live:** `Host: felhom.demo-felhom.eu` → **200**; `Host: 1.2.3.4` →
|
|
**404** (catch-all). External browser access (`felhom.demo-felhom.eu`) needs the **Cloudflare
|
|
tunnel**, which is **unconfigured on a bare bootstrap** (`cf_tunnel_token` empty) — same slice-10
|
|
onboarding gap as below; not separately testable without the tunnel/DNS.
|
|
2. **`/api/host-metrics` (slice 9)** — `{ok:true}` HTTP 200. **Cross-checked against the host:**
|
|
memory_total **exact** (16537989120 = pvesh), memory_used ~match (2.49 vs 2.43 GB), loadavg same
|
|
ballpark, uptime match (324255 vs 324283), disk match (93.9 GB / 10.5% vs `df` 94G/12%),
|
|
felhom-usb match (915.8 GiB / 0.87%, SMART 31°C). **`cpu_temp_c` is a real value** — read 46 live,
|
|
matching sysfs `x86_pkg_temp=44`/coretemp max 46 (`sensors` is **not installed**; the agent sources
|
|
temp from coretemp/thermal, not the `sensors` CLI). An earlier reading of 59 was a genuine transient
|
|
load peak during the golden-build+provision.
|
|
3. **`/api/disks` (8C proxy)** — `{ok:true}` HTTP 200, 4 devices with data-bearing flags;
|
|
**felhom-usb flagged `data_bearing:true` (reason "device is mounted", `/dev/sdb1`)**.
|
|
|
|
### BREAK 3.2 — storage auto-discovery (by-design consequence of 8C de-privileging)
|
|
The 1TB HDD does **NOT** appear under Settings → Storage Paths. The de-privileged container sees **no
|
|
host storage mounts** (`df` inside shows only the bootstrap bind, its own disk, udev; `/mnt` empty;
|
|
logs: "no storage paths registered", "stat /mnt/sys_drive: no such file or directory"). This is correct
|
|
for the 8C model (mounts limited to bootstrap+data+docker.sock). The HDD is instead surfaced via the
|
|
**agent host-metrics storage view** (felhom-usb appears there with capacity + SMART). The legacy local
|
|
`discoverHDDPaths` path is effectively **vestigial** in v0.39.0 — worth retiring or repurposing onto the
|
|
agent-sourced storage list.
|
|
|
|
### BREAK 3.3 — deploy→run→remove an app: not exercisable on a bare bootstrap
|
|
`/api/stacks` → `{ok:true,data:[]}` (empty catalog). Catalog templates come from a synced
|
|
`catalog-cache/templates`, populated by git/assets sync — **disabled on a bootstrap-seeded controller**
|
|
(manual mode, no repo URL; the seed sets only identity/hub/local-api). No apps → nothing to deploy
|
|
(`POST /api/stacks/filebrowser/deploy` → 400 "invalid request body"; even with a body the slug isn't in
|
|
the catalog). Catalog configuration is **slice-10 (hub desired-state)** territory.
|
|
|
|
### BREAK 3.5 — hub reporting stays DOWN (HTTP 401): host-key vs customer-key gap
|
|
The controller's startup push and all 3 retries got **HTTP 401** from the hub
|
|
(`[report] Push failed: HTTP 401`), reproduced directly (`POST https://hub.felhom.eu/api/v1/report`
|
|
with the baked key → `Unauthorized` / 401). **Root cause (code-traced + DB-confirmed):**
|
|
- The hub's `POST /api/v1/report` authenticates a **customer-scoped** key — `checkAuthCustomer` →
|
|
`GetCustomerConfigByAPIKey` against the `customer_configs` table, then enforces
|
|
`authCustomerID == payload.CustomerID` (`hub/internal/api/handler.go:74-92, 208-234`).
|
|
- Provision baked the **agent's HOST key** (`hub.api_key`, keyed on `host_id=demo-felhom-01` in the
|
|
separate `hosts` table) into the guest's bootstrap. Host keys and customer keys are **distinct tables
|
|
/ code paths**.
|
|
- Hub DB confirms: a `demo-felhom` customer row exists (with its **own** dashboard-generated api_key),
|
|
a `demo-felhom-01` host row exists, and the baked key appears **once** (as the host key). So the
|
|
controller presents the host key → `GetCustomerConfigByAPIKey` returns nil → **401**.
|
|
|
|
The `demo-felhom` hub row therefore **stays DOWN/stale** — the freshly-provisioned controller can never
|
|
report ONLINE until the bootstrap carries the **customer-scoped** api_key (or provision creates/fetches
|
|
the customer config key, or the hub accepts host keys for customer reports). This is a **cross-component
|
|
provisioning gap (slice-10 onboarding)**, not a controller bug. *(Reported value differs from the
|
|
brief's "v0.34.0" — the DB shows last reports up to v0.28.8; immaterial, the row is stale either way.)*
|
|
|
|
---
|
|
|
|
## Step 4 — Disk proxy (API-level only; NO destructive op) — INVARIANT PROVEN
|
|
|
|
1. **List** — `/api/disks` `{ok:true}` with per-device data-bearing flags (above).
|
|
2. **Data-bearing format → refusal** — hit the agent **directly** (dodges the controller's CSRF):
|
|
`POST https://192.168.0.162:8443/disks/format` with the guest's `local_api.token`, body
|
|
`{"vmid":9200,"device":"/dev/sdb1","fstype":"ext4"}` (felhom-usb — mounted, data-bearing). Result:
|
|
|
|
**HTTP 403** — `{ "formatted": false, "data_bearing": true, "reason": "device is mounted",
|
|
"pending_op": { "op":"storage_wipe", "host_scope":"demo-felhom-01",
|
|
"durable_id":"byid:wwn-0x5000039ddb108568-part1", "fstype":"ext4" },
|
|
"error": "device is data-bearing — format requires an operator signature (pending_signature)" }`
|
|
|
|
The agent **inspected the device itself** (`data_bearing:true`, reason "device is mounted"),
|
|
ignored any caller claim, refused with `pending_signature`, and surfaced the durable-id-bound op to
|
|
sign. **The disk was untouched** (post-test: `/dev/sdb1 ext4 915.8G, 8G used`, still mounted). No
|
|
operator signature was ever passed. The controller maps this 403→409 (`agentapi.ErrFormatRefused`,
|
|
already unit-tested). ✓
|
|
|
|
---
|
|
|
|
## Step 5 / 6 — Orphan-template cleanup + gate + push
|
|
|
|
- **Deleted** (re-confirmed unreferenced first — `grep -rn` over `internal/` matched only the
|
|
templates' own `{{define}}` lines): `internal/web/templates/{storage_init, storage_attach, migrate,
|
|
migrate_drive, restore}.html`. Embed is a glob; 14 templates remain.
|
|
- **Noted, not deleted** (dead-but-harmless): `NotifyCrossDriveCompleted`/`NotifyCrossDriveFailed`
|
|
(`notify/notifier.go:353,359`, no callers) + a vestigial `crossdrive_failed` notification toggle
|
|
(`web/handlers.go:937`) + restic config fields/comments. Flagged for a future dedicated cleanup.
|
|
- **Version:** v0.39.0 → **v0.39.1** (CHANGELOG entry added; version is ldflags-injected, applied at
|
|
the next build). Source-only — no re-bake this session.
|
|
- **Gate:** `go build ./...` **OK**; `go test ./...` **green** (agentapi, bootstrap, quiesce).
|
|
- **Commit:** `6e77bea` (the template deletion + this report), pushed to `main` (`d8d1e17..6e77bea`).
|
|
No working UI feature lost (the deleted pages were already unreachable — removed routes).
|
|
|
|
---
|
|
|
|
## What broke / what's missing (the headline)
|
|
|
|
| # | Item | Severity | Nature |
|
|
|---|---|---|---|
|
|
| 2a | Local-API channel 401 until `felhom-agent` restart after provisioning a live-daemon host | **Medium** | Provision doesn't make the running daemon reload its token store. Workaround: restart the daemon (done). Needs a provision→daemon reload signal or per-request store lookup. |
|
|
| 3.5 | Hub report 401 — bootstrap bakes the **host** api_key, but `/api/v1/report` needs the **customer** api_key | **Medium/High** | Cross-component provisioning gap (slice-10 onboarding). Controller stays DOWN on the hub until fixed. |
|
|
| 3.3 | No app catalog on a bare bootstrap (git/assets sync disabled) — deploy not exercisable | Expected (slice-10) | Catalog/desired-state comes from the hub later. |
|
|
| 3.2 | Legacy Storage-Paths auto-discovery finds nothing (de-privileged container has no host mounts) | Expected (8C) | HDD is correctly surfaced via agent host-metrics instead; retire/repurpose the legacy path. |
|
|
| 3.1 | External browser access (felhom.demo-felhom.eu) needs the Cloudflare tunnel (`cf_tunnel_token` empty) | Expected (slice-10) | Host-gate routing itself verified live (200 vs 404). |
|
|
| 2b | Bootstrap seed-log line absent | Cosmetic | Functionally correct; logging-only. |
|
|
| infra | Supplied Gitea token lacks registry/package scope; build-golden default tag stale (`:v0.35.0`) | Low | Used the build-server credential for the re-bake; flagged both for Viktor. |
|
|
|
|
**Worked cleanly:** golden re-bake, provision + reboot deploy, configured-not-setup bootstrap, 0600
|
|
bootstrap.json/controller.yaml, container de-privileging (Privileged=false, 3 mounts), no-registry-pull,
|
|
all 5 UI pages + slice-9 host card, host-metrics (cross-checked, real cpu_temp), `/api/disks`, and the
|
|
**8C data-bearing format-refusal invariant (403, disk untouched)**.
|