admin
88362dac0a
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
522 lines
47 KiB
Markdown
522 lines
47 KiB
Markdown
# CONTEXT.md — Project Memory
|
||
|
||
> This file serves as persistent project memory across Claude Code sessions.
|
||
> It replaces the auto-generated "Memory" from the claude.ai Project.
|
||
> **Update this file at the end of each working session** with current state,
|
||
> recent decisions, and anything the next session needs to know.
|
||
>
|
||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||
|
||
Last updated: 2026-07-03 (docs: CLAUDE.md refreshed — stable orientation; runbooks live in the felhom skills)
|
||
|
||
> **2026-07-05 — v0.99.0 restore-path fixes (TASK C1): drill findings F1/F3/O4 RESOLVED.**
|
||
> F1: `GET /api/backup/snapshots` implemented (`backup.ListRestorePoints`) — the restore panel
|
||
> populates and the restore button enables. F3: `runVolumeDumps` wired into the nightly/manual
|
||
> backup run (volume gate BEFORE DumpAppVolumesSafe; before unit capture). O4: unrecoverable
|
||
> resettable secrets get a generated replacement (`GenerateSecretForField` + `SetSecretGenerator`
|
||
> seam); data-key gate untouched. **F2 (one-click in-place class-C restore) remains OPEN → TASK C2.**
|
||
> O4 residual: restored volume tar with an OLD credential hash may still need a manual in-DB reset.
|
||
|
||
> **2026-07-04 — app-data restore drill** → see `felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md`.
|
||
> Keep-side restore proven live on 9201 (class-A DB replay + fail-closed data-key gate + non-destruction). **F1 (HIGH): UI restore is dead — `/api/backup/snapshots` has no handler, so the restore button never enables.** F2: no one-click in-place class-C (HDD bind-mount) restore. F3: named-volume data never backed up (`DumpAppVolumes*` has no caller). *(F1/F3/O4 resolved in v0.99.0, see above.)*
|
||
|
||
> **2026-07-03 — CLAUDE.md slimmed to stable orientation** (full package map, verified 9201 deploy
|
||
> summary, no version-pinned state). Deep runbooks/design/testing doctrine now in the personal
|
||
> skills `felhom-build-deploy` / `felhom-ui-design` / `felhom-testing` (source: `felhom.eu/skills/`).
|
||
|
||
> **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / patterns / traps / seams, code-verified). Check it before writing new code; update it in the same commit that adds/changes a shared helper (maintenance rule now in CLAUDE.md).
|
||
|
||
> **2026-07-02 — v0.98.0 (deployed on 9201): Tárhely IA follow-up.** User feedback on D1: NAS-add and
|
||
> local-drive enrollment buttons sat side by side — confusing. `/storage` split into two subpages
|
||
> under the Tárhely nav item (nested sub-links, `.nav-links-nested`): **/storage = Meghajtók**
|
||
> (drives + agent view + wizards + manual add) and **/storage/network = Hálózati tárhely (NAS)**
|
||
> (NAS-megosztások list + add form; own openDialog copy). `networkStoragePageData` (page key
|
||
> `storage-network`) split out of `storagePageData`. No API/storage-semantics change.
|
||
|
||
> **2026-07-02 — v0.97.0 (deployed on 9201): TASK-D1 — settings split + Tárhely page (IA only).**
|
||
> The 1451-line `settings.html` monolith is split into four pages: `/storage` (new main-nav
|
||
> **Tárhely**: Adattárolók + NAS + unified agent drive view + wizard entry), `/settings` (Rendszer:
|
||
> konfig, verzió/frissítés, vezérlő+kiszolgáló újraindítás), `/settings/notifications` (Értesítések
|
||
> + Alkalmazás-email), `/settings/security` (Jelszó, Földrajzi korlátozás, Vészhelyzeti információk).
|
||
> Sidebar gains the Tárhely item + a "Beállítások" group with three sub-links. **No storage/agent/API
|
||
> behavior change** — pages moved, logic didn't; every `/api/storage/*` endpoint + POST action
|
||
> unchanged. **Decisions:** (a) storage flashes + wizards moved to `/storage`; old
|
||
> `/settings/storage/{init,attach}` **301** to `/storage/{init,attach}`. (b) The two duplicated drive
|
||
> views MERGED: registry cards render server-side, then JS enriches each connected user-data card in
|
||
> place from the agent `/api/disks` (role tag, durable-id, agent actions) joined on mount path, plus
|
||
> two groups — **Rendszermeghajtók** (system/backup, read-only, lock tag, NO actions) and **Nem
|
||
> regisztrált** (register only); agent-down → one warn note, registry cards still render. (c) Every
|
||
> native `confirm()`/`prompt()` on the four pages now routes through a light `.confirm-overlay`
|
||
> (`openDialog`; texts verbatim; type-to-confirm kept only where it existed) — the D0 tab-freeze is
|
||
> gone. **Contract for future UI:** new templates must pass `scripts/template_id_gate.py` (every JS
|
||
> element-ID resolves in its own template) and `scripts/emoji_gate.py` (0 emoji — the D0 grep gate
|
||
> false-negatived multibyte emoji on Windows; 8 survivors removed). `.badge-lock`/`.lock-ico` CSS
|
||
> deleted; other `.badge*` still used on secondary pages (D2 to finish). **Next: D2** (badge→tag on
|
||
> backups/secondary pages; storage.html is still ~700 lines — NAS-add partial candidate). NOT
|
||
> live-validated: agent-down degradation (don't stop the live agent — static/unit only); destructive
|
||
> storage ops via the moved overlay paths (endpoints unchanged; supervised session).
|
||
|
||
> **2026-07-02 — v0.96.0 (deployed on 9201): TASK-D0 — design system v2 re-skin (appearance only).**
|
||
> The whole customer UI (dashboard AND setup wizard) now renders in the approved v2 language: navy
|
||
> token palette (`--bg-0/1/2, --line, --text-1/2/3, --blue, --warn, --crit`), single 2px radius, no
|
||
> shadows; **exception-based status color** — nominal is blue/neutral, amber/red only on deviation;
|
||
> zone bars → the hairline `.meter` (neutral 70/85 ticks, warn/crit flag „Fogyóban a hely" /
|
||
> „Kritikusan kevés hely"); pills → `.tag` / `.metarow`. **Decisions:** (a) `stateColor` semantics
|
||
> changed — **stopped/exited = neutral, NOT red** (operator-approved; failures surface via unhealthy
|
||
> + alerts); restarting = warn; outputs are now `run/progress/warn/neutral/off` and
|
||
> `nominal/warn/crit` — any new template MUST use these suffixes (truth tables guarded by unit tests,
|
||
> `stateLabel` copy frozen byte-identical). (b) Fonts (PJS + JBM variable woff2, latin+latin-ext) and
|
||
> a 30-icon Lucide sprite are **vendored in the binary** (`/static/fonts/`, `templates/icons.html`)
|
||
> — no CDN; `fonts.googleapis.com`, `box-shadow`, 999px and the emoji set are banned strings (grep
|
||
> gate in REPORT §5, 143→0). (c) Setup wizard `handleCSS` bug fixed: it read a dataDir-derived path
|
||
> that never exists in the container → always served minimalCSS in production; now serves embedded
|
||
> `web.StyleCSS()`. (d) Found+fixed pre-existing v0.93.0 bug: `/backups` 500'd once an off-box backup
|
||
> had run (`OffboxTarget.LastRun` is an RFC3339 string fed to `timeAgo`; new `timeAgoStr`).
|
||
> Canonical reference: `felhom.eu/documentation/design/design-system.md`. **Next: D1** (settings IA
|
||
> split + Tárhely main-nav promotion; also migrate the remaining native `confirm()` dialogs to the
|
||
> type-to-confirm overlay and finish badge→tag markup on secondary pages). NOT live-validated: setup
|
||
> wizard visuals (unit-only), warn/crit meters on real hardware (demo healthy).
|
||
|
||
> **2026-07-01 — v0.95.0 (deployed on 9201): Impl-2b — raw-drive enrollment wizards.** Both enrollment
|
||
> wizards (`storage_init.html` / `storage_attach.html`) now fetch candidates from the agent's raw-device
|
||
> scan `GET /api/disks/candidates` (proxy `agentDiskCandidatesHandler` → agent Impl-2a) instead of the
|
||
> `Observe()`-based `/api/disks`, so a brand-new (non-PVE-storage) drive is discoverable + enrollable.
|
||
> `agentapi.ListCandidates` + `resolveEnrollUUID` (a raw candidate isn't in `/disks` → resolve its
|
||
> fs-UUID from the scan). Live end-to-end on felhom-pve: raw `/dev/sdd` (SD card) → wizard → format (Impl-1
|
||
> guard) / attach → mount → bind → intent → guest-bind → **"Aktív"** in the UI (3rd drive, no false
|
||
> detach). Needed a chain of AGENT fixes (v0.56–0.58: durableIDForMount / ReassertGuestBinds / HostReader
|
||
> wiring / /disks GuestPath+BoundUnderParent for registry rows) — the whole enroll+tracking machinery had
|
||
> assumed a PVE-storage drive. **Follow-ups:** `runStorageInit` should poll the agent's detached-format
|
||
> status (`/disks/format/status`) so a SLOW-device format completes in one flow (pre-existing); Impl-3
|
||
> shared-box operator format gate.
|
||
|
||
> **2026-06-30 — v0.94.0 (deployed on 9201): pull-based config-refresh.** The hub report ACK now also
|
||
> carries a per-customer **`config_version`** (hub v0.26.0; a stored counter bumped on every config save).
|
||
> `OnPushResponse` → `ConfigRefresher.Reconcile` (`internal/report/config_refresh.go`): on a change vs.
|
||
> `settings.applied_config_version`, `bootstrap.RefreshConfig` re-pulls `controller.yaml` (re-merging
|
||
> `local_api` from bootstrap.json; overwrites controller.yaml, never settings.json) → record → graceful
|
||
> self-restart (`api.GracefulSelfRestart`). First-run records baseline (no restart); unchanged = no-op
|
||
> (no storm); failed pull keeps config + retries. This is the box-pulls-config replacement for the hub's
|
||
> retired inbound "Push Config" — the hub never connects into the box. **Companion hub v0.26.0** also
|
||
> retired Trigger Update / Pull Config / Show Diff and dropped geo-disable's inbound notify (kept the
|
||
> hub→Cloudflare WAF removal), and fixed the stale `docker-setup.sh` setup command → host-install.
|
||
> Live-validated on 9201: first-run baseline (applied=1, no restart) → DB bump 1→2 → re-pull+self-restart
|
||
> (session_secret rotated, local_api preserved, RestartCount 0→1) → 4 cycles no-loop → apps stayed up.
|
||
|
||
> **2026-06-29 — Self-health arc complete (agent v0.48.0 + hub v0.22.1).** The hub now proactively
|
||
> watches every agent's served local-API **leaf fingerprint** fleet-wide: the agent reports
|
||
> `leaf_fingerprint` on its host report (v0.48.0), and the hub `HostLeafChecker` (`monitor/host_leaf.go`)
|
||
> raises `host_leaf_changed` when it changes (trust-on-first-report baseline; empty fp = unknown, never
|
||
> alerts; reads from report_json, no migration; hub-generated so no allowlist change). Independent of the
|
||
> controller channel-check. Live: a leaf regen raised `host_leaf_changed` (`82078fab…→60b5974d…`) AND the
|
||
> controller's `agent_channel_pin_mismatch` — complementary. NOTE: v0.22.0 shipped the checker but the
|
||
> main.go wiring never applied (concurrent file-touch); the LIVE TEST caught it (no event on regen) →
|
||
> fixed in v0.22.1. The original silent-multi-day-outage incident class is now caught from THREE angles
|
||
> (agent capabilities v0.44.0 / controller channel-check v0.90.0+F2 / hub leaf-fp v0.22.x) AND prevented
|
||
> (agent v0.46.0 loud-regenerate + host-install --preserve-state-from). Backlog: the served-fp-vs-pinned-fp
|
||
> authoritative cross-check; the test-run coverage gaps (capability→hub live, host-reboot doubling).
|
||
|
||
> **2026-06-29 — v0.91.0: F2 (channel-health born-down alerting).** The v0.90.0 channel-health check
|
||
> alerted only on a live up→down transition; a channel broken at **startup/reseed** (e.g. the
|
||
> controller boots right after a leaf regen → first observation is `pin_mismatch`) was dashboard-only,
|
||
> no operator email — forever. Fixed with an `alerted` flag in `channelhealth/checker.go`: a confirmed
|
||
> down coming from up/unseeded OR a reason-change re-arms then alerts once; steady-down no dup; recovery
|
||
> re-arms; debounce intact (transient born-down still N≥2; healthy first-obs silent). Live-validated:
|
||
> controller restarted into a regenerated-leaf channel → `[channel] DOWN (unseeded->down:pin_mismatch)`
|
||
> + `Event pushed: agent_channel_pin_mismatch` (~60s) — the capstone's missing half. Companions: hub
|
||
> v0.21.0 (same fix for HostCapabilityChecker/HostStalenessChecker — seed only healthy, born-degraded/
|
||
> stale emits on first Check, cooldown dedups), agent v0.46.0 (EnsureLeaf loud-WARNs a REGENERATED leaf
|
||
> + host-install `--preserve-state-from`/populated-host guard = the prevention for the original incident
|
||
> class). Self-health story now complete: agent watches its capabilities, controller watches its link,
|
||
> detection works at startup, and a reinstall can preserve the leaf. Backlog: F1 swap-rollback
|
||
> RestartCount hardening; full live reinstall (supervised); hub-side leaf-fp fleet comparison.
|
||
|
||
> **2026-06-29 — v0.90.0: controller→agent channel health-check (self-health slice).** A ~60s
|
||
> scheduler job (`internal/channelhealth`) probes the local-API channel via the PRODUCTION memoized
|
||
> client (`Server.ProbeAgentChannel` → `s.agentClient()` + GET /storage — NOT a fresh client, per the
|
||
> spike: self-heals, mirrors the disk UI, no transport leak). Classifies failures (spike Q1 map:
|
||
> pin_mismatch/unauthorized/unreachable/timeout/misconfigured/construction_error/unknown), **debounces**
|
||
> transient reasons (refused/timeout need N≥2 consecutive so the ~1s agent-restart blip doesn't page;
|
||
> pin/401/DNS/construction alert first-obs), seeds the first observation silently, and on a transition
|
||
> emits an **English operator-only** event (`Notifier.NotifyAgentChannelDown/Recovered`) + a Hungarian
|
||
> dashboard banner (`AlertManager.SetAgentChannelAlert`). Closes the R1-incident gap (channel was only
|
||
> checked once at startup). **Required a hub change** (felhom-hub v0.20.0): the `agent_channel_*` event
|
||
> types were rejected by the hub's `allowedEventTypes` allowlist (HTTP 400) — controller-pushed events
|
||
> are gated, unlike the hub-generated host_* ones. Live-validated: transient restart → no alert;
|
||
> sustained stop → debounce(1/2) then DOWN transition + dashboard banner; start → recovered. No agent
|
||
> change. Self-health story now: agent watches its own capabilities (v0.44.0) + controller watches its
|
||
> link to the agent (this). Backlog: hub-side leaf-fp comparison.
|
||
|
||
> **2026-06-29 — OPS (no code change): controller↔agent TLS leaf-pin mismatch RESOLVED via R1.**
|
||
> The 2026-06-28 root→non-root agent migration regenerated the agent's local-API leaf
|
||
> (`60b5974d…`→`ced34036…`) and truncated its token store, so the in-guest controller's pinned
|
||
> `agentapi` client failed every call (`TLS pin mismatch`) — disk UI dead AND the quiesce
|
||
> app-consistent-backup loop dead. Fix: restored the pre-migration `local-api.{crt,key}` **+**
|
||
> `local-tokens.log` from `/root/agent-backup-bundle-test/aside-var-lib/` on felhom-pve (R1 RUNBOOK,
|
||
> supervised). Agent now serves `60b5974d…` again (== controller pin + bootstrap.json seed) and the
|
||
> restored token store's 9201 entry matches the controller's current `local_api.token` — **zero
|
||
> in-guest change.** Gates passed: leaf loaded (not regenerated), wire fp `60b5974d…`, authenticated
|
||
> `GET /disks` 200/5-disks (bogus token 401), and the next quiesce cycle ran a real backup job.
|
||
> Rollback point kept at `/root/agent-rollback-20260629-150137/`. **OPEN (carried forward):** the
|
||
> migration/Day-0 path must carry these three files into the new `felhom-agent`-owned data dir (or
|
||
> treat their loss as a re-bootstrap trigger), else any re-migration reintroduces this. Full record:
|
||
> `felhom.eu/documentation/audits/SPIKE-agent-pin-mismatch-2026-06-29.md` (§9 Outcome).
|
||
|
||
> **2026-06-27 — v0.86.0 (deployed on 9201): Phase 2 managed updates — controller-version FLOOR.**
|
||
> On top of the Phase 1 opt-in "update to latest" button, the controller now honors an operator-enforced
|
||
> **minimum version (FLOOR)** delivered on the hub **report ACK** (`min_controller_version` +
|
||
> `latest_version`; `internal/report/pusher.go` `PushResponse`). `OnPushResponse` →
|
||
> `updater.SetFloor()` + `updater.MaybeAutoUpdate()` (rides the report cycle — no new timer). Below the
|
||
> floor → **auto-update to the floor** (reuses Phase 1 `performUpdate`: in-guest pull → agent swap →
|
||
> rollback; `initiatedBy="auto-floor"`). At/above floor → nothing (does NOT chase latest — that's the
|
||
> button). Guards: dev/no-agent/backup → skip; floor must be pullable (floor ≤ latest; floor>latest →
|
||
> warn+noop); one attempt per below-floor condition (in-mem + persisted state) → no flapping. Floor source
|
||
> + operator UI are hub-side (felhom-hub v0.15.0: per-customer override + global default + report ACK).
|
||
> **No agent change** (reuses Phase 1 `POST /controller/swap`). Day-0 now ships current (golden rebuilt at
|
||
> 0.85.1) AND stays current (floor) — the fleet-currency story is closed. Live: dogfood 0.85.1→0.86.0 via
|
||
> the button, then floor 0.87.0 → auto 0.86.0→0.87.0 (no click).
|
||
|
||
|
||
> **2026-06-26 — v0.84.0 (deployed on 9201): show an app's auto-generated first-login on its page.**
|
||
> General, catalog-driven mechanism: `.felhom.yml initial_credentials: {file, format json|regex|plain,
|
||
> username_key/password_key | username_pattern/password_pattern, note}`. The controller reads the file
|
||
> **live** from the container (`internal/stacks/initialcreds.go` `ReadInitialCredentials` + pure
|
||
> `parseInitialCreds`, unit-tested), never persists it, and renders a "Kezdeti belépési adatok" card on
|
||
> `/apps/{slug}` (masked password + reveal/copy). First consumer: crafty-controller (Crafty writes a
|
||
> random admin password to `/crafty/app/config/default-creds.txt`). Live-verified on 9201: card shows
|
||
> username `admin` + the real extracted password. Reuse for any future self-seeding app. Security: same
|
||
> exposure class as the existing post-deploy reveal / default_creds card — relies on the prod dashboard
|
||
> being auth-gated (demo public-unauth is the separate tracked issue). Backlog idea unchanged: a
|
||
> `backend_scheme` hint for TLS backends (v0.83.0 line).
|
||
|
||
> **2026-06-26 — v0.83.0 (deployed on 9201): scoped Traefik backend transport for self-signed HTTPS apps.**
|
||
> Crafty is the **first/only catalog app with an HTTPS backend** (self-signed TLS on `:8443`, no plain-HTTP
|
||
> port). The crafty healthcheck fix un-withheld its route, exposing a pre-existing 502 (Traefik proxied
|
||
> HTTP to the HTTPS backend). Since traefik v3 forbids `insecureSkipVerify` via Docker labels, the
|
||
> controller now renders a file-provider dynamic file `dynamic/serverstransports.yml` defining a **named**
|
||
> `insecure-skip-verify` transport (`infra.RenderServersTransports` + `stacks.ensureServersTransports`,
|
||
> written from `EnsureBaseStack` outside the traefik-running guard, write-if-changed, hot-loaded). Apps opt
|
||
> in per-service via catalog labels `scheme=https` + `serverstransport=insecure-skip-verify@file`.
|
||
> **No global `insecureSkipVerify`** — verification stays ON for every other backend (scoped Option B).
|
||
> Pattern to reuse for any future self-signed HTTPS backend. Live-verified: `minecraft.demo-felhom.eu`
|
||
> 502→302; filebrowser (HTTP) unaffected. Backlog: a `.felhom.yml backend_scheme` hint so the catalog
|
||
> convention sets these labels instead of hand-adding.
|
||
|
||
> **2026-06-22 — v0.76.0 (deployed on 9201): three campaign-#3 hardening fixes.**
|
||
> - **S1**: corrupt `settings.json` no longer crash-loops — `save()` writes a last-known-good `.bak`
|
||
> (after the primary rename); `Load()` recovers from `.bak`, else preserves the corrupt file as
|
||
> `*.corrupt-<ts>` + safe defaults. New `Settings.LoadWarning` → dashboard banner. (`main.go` Fatalf
|
||
> now only on the IO-unreadable path.)
|
||
> - **F2**: `web.validStackName` gates `backupRestoreHandler` + `apiExportStart` (reject `/ \ .. NUL`)
|
||
> before any restore/export — closes the traversal defense-in-depth gap (no escape had occurred, but
|
||
> it relied on downstream map-lookups).
|
||
> - **S3**: `quiesce.readMarker` logs `[WARN]` + quarantines a corrupt marker to `*.corrupt-<ts>`
|
||
> instead of silently dropping it.
|
||
> All three live-validated on 9201 (truncate settings → recover from .bak no crash-loop; traversal
|
||
> restore → rejected, /etc intact; corrupt quiesce marker → quarantined). Tests + red-proofs.
|
||
> Still-open campaign-#3 deferrals (NOT done): time-chaos on a dedicated VM, host reboot (supervised),
|
||
> `.fab` import compose-fuzzing. See `felhom.eu/documentation/tests/test-campaign-3-2026-06-22-findings.md`.
|
||
|
||
Last updated: 2026-06-22 (v0.75.0 — gate userdata MkdirAll on a live mountpoint)
|
||
|
||
> **2026-06-22 — v0.75.0 (deployed on 9201): userdata MkdirAll gated on a live mountpoint.**
|
||
> Both `MkdirAll`-into-`<drive>/userdata` sites — the deploy belt (`stacks.ensureUserdataMounts`) and
|
||
> the FileBrowser sync (`web.syncFileBrowserMounts`) — now skip when an **external** drive root (under
|
||
> `StableParentDir=/mnt/felhom-drives`, `!= sysDataPath`) is **not a live mountpoint** (`system.IsMountPoint`).
|
||
> Closes the campaign-#2 hazard where a drive-absent window produced `mkdir …/userdata: permission denied`
|
||
> + transient `Created` flapping AND could write app data onto the guest **rootfs** (shadowed when the
|
||
> drive returns). The app is held by `planDriveGates` instead. System/local path is never gated. New
|
||
> `Manager.isMountPoint` seam + pure `web.skipFileBrowserPath` helper for tests. Live-proven via a C6
|
||
> re-run: 0 permission-denied, 0 shadow dirs on the rootfs, apps recover on reconnect.
|
||
>
|
||
> **Boot-ordering design note (NOT implemented — decide separately):** the *boot-time* `mkdir …
|
||
> permission denied` is a different cause — **docker's** boot-restore auto-starts drive-backed
|
||
> containers (`restart: unless-stopped`) before the agent mounts the drives, so docker (not the belt)
|
||
> tries to create the bind source; `planDriveGates` recovers them after mount convergence (why a reboot
|
||
> ends healthy). Options: (A) accept + suppress (apps self-heal; lowest risk; maybe downgrade the
|
||
> boot-window log level) — recommended now; (B) drive-backed app containers don't docker-auto-start at
|
||
> boot (restart policy `no`/`on-failure`) so the controller's gate is the sole starter after mounts
|
||
> converge — cleaner but bigger (must cover crash-restart too). See
|
||
> `felhom.eu/documentation/tests/test-campaign-2-finding1-recovery-diagnosis.md` (boot-ordering observation).
|
||
|
||
Last updated: 2026-06-22 (v0.74.0 — controller→agent connection-leak fix)
|
||
|
||
> **2026-06-22 — v0.74.0 (deployed on demo guest 9201): fixed the controller→agent connection leak.**
|
||
> `Server.agentClient()` built a new `agentapi.Client` (new bare `http.Transport`, `IdleConnTimeout:0`)
|
||
> per call → one leaked idle ESTABLISHED socket per agent call to `192.168.0.162:8443`, exhausting the
|
||
> ephemeral source-port range after ~5 days of controller uptime → EADDRNOTAVAIL, which had taken down
|
||
> storage UI + host-metrics + whole-guest backup (found in the 2026-06-22 unattended campaign). Fix:
|
||
> memoize ONE shared client via `sync.Once` + harden the Transport (`MaxIdleConnsPerHost:2`,
|
||
> `IdleConnTimeout:90s`). Live-proven: idle sockets to `:8443` stay flat at 2 across a 120-call burst
|
||
> (pre-fix grew ~1/call → 132). Agent/firewall untouched. Diagnosis + campaign findings live in
|
||
> `felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-*.md`. **Separate open item:** the
|
||
> defense-in-depth host firewall rule scoping `:8443` to the guest bridge subnet is still absent
|
||
> (pve-firewall disabled) — to be closed independently.
|
||
|
||
Last updated: 2026-06-13 (v0.60.0 backlog-Medium cleanup)
|
||
|
||
> **Live version: controller v0.60.0** (deployed on demo guest 9201), agent **v0.30.0** (AGENT-001 deployed), hub **v0.11.0**.
|
||
> **NOTE:** the long-form sections far below this banner are stale (last full pass ~v0.16.1). Current
|
||
> state is the dated entries immediately below + `CHANGELOG.md` + the now-authoritative **central docs at
|
||
> `felhom.eu/documentation/controller/`** (code-verified) + auto-memory `MEMORY.md`.
|
||
>
|
||
> **2026-06-13 — v0.60.0 backlog-Medium cleanup (BUGHUNT reconcile):**
|
||
> - **M25** (Server.integrationMgr data race — constructor goroutine vs post-construction `SetIntegrationManager`):
|
||
> **FIXED** via `atomic.Pointer`; `-race`-verified. **M4/M5/M6** verified already FIXED (no action).
|
||
> - **M18** (dump re-validation every 5 min — perf) and **M19** (`deriveStackName` misattribution — low
|
||
> incidence) verified LIVE but cross-package-entangled → branches `fix/m18-dump-validation-cache` /
|
||
> `fix/m19-stackname-crossref` (notes + fix plan, PENDING REVIEW, not deployed).
|
||
>
|
||
> **2026-06-13 — v0.59.0 audit fixes + documentation centralization:**
|
||
> - Fixed the validated 2026-06-13 audit findings (records: `felhom.eu/documentation/audits/`):
|
||
> **CTRL-001** (`.fab` import path traversal — manifest segment validator, fail the parse);
|
||
> **CTRL-T2-1** (ghost-deployed on crash — `app.yaml` now persists `deployed:false` until `compose up -d`
|
||
> succeeds, flipped true only on success; in-memory flag still true during pull for UX);
|
||
> **H10** (plaintext secret on encrypt failure — `SaveAppConfig` now fail-closed, returns an error);
|
||
> **M2** (misleading lock on init-only `SetStackProvider` removed). All with regression tests.
|
||
> - **AGENT-001** (wrong-disk wipe TOCTOU) fixed on agent branch `fix/agent-001-wipe-durable-reresolve`
|
||
> — PENDING REVIEW, NOT deployed (supervised merge+golden-rebake reserved; see `AGENT-001-FIX-NOTES.md`).
|
||
> - Built + deployed v0.59.0 to demo guest 9201 (bootstrap mechanism); verified healthy, dashboard 200.
|
||
> - Documentation centralized under `felhom.eu/documentation/` (controller subtree + audits + top index),
|
||
> code-verified; `controller/README.md` banner points there; this CONTEXT banner refreshed.
|
||
>
|
||
> **2026-06-13 — v0.58.0 OS/Docker-data split prevention layer (Phase 2; Phase 1 = agent v0.29.0):**
|
||
> - OS rootfs + Docker data split onto separate local-lvm volumes (golden bakes 32G rootfs + 256G
|
||
> /var/lib/docker, backup=1, **overlay2** so images live on the data vol). Infra protected by
|
||
> PREVENTION not placement: `system.GetDockerVolumeHeadroom()` (statfs "/" = data vol) reserves
|
||
> max(5GB,10%); `deployStack` refuses HTTP 507 below buffer; deploy.html banner+disable; monitor
|
||
> (warn80/crit90) watches the same vol; log rotation baked into the golden daemon.json.
|
||
> - Live-validated by DESTROYING + RE-PROVISIONING 9201 from the split golden: split layout, overlay2,
|
||
> images on data vol, lean rootfs, OS isolation (data vol 100% → rootfs healthy), deploy gate 507,
|
||
> ActualBudget volume on data vol, hub config pull, external access via CF. RomM/USB re-enroll = the
|
||
> documented final restore step (RomM data safe on host USB). See memory [[os-data-split]].
|
||
>
|
||
> **2026-06-13 — v0.57.0 UI fixes (Part A of the UI-fixes/storage-spike spec):**
|
||
> - A1: fixed the RIGHT storage list — `#host-storage-bars` (the JS-filled, agent-PVE-storage list:
|
||
> `local`/`local-lvm`/`felhom-pbs`/`felhom-usb`), which reordered on every poll. Now
|
||
> `enrichHostStorageTargets` sorts `/api/host-metrics` server-side + adds friendly Hungarian
|
||
> labels/purpose. Display-only — PVE storage ids never renamed. (v0.56.0's 4C had sorted the OTHER,
|
||
> server-rendered user-data list.)
|
||
> - A2: per-app Tier-2 config panel at `GET/POST /stacks/{name}/backup`; the dead-end "Beállítás" button
|
||
> (was → deploy page) is repointed there. Pin a target drive / toggle Tier 2 off; prefs
|
||
> (`UserDisabled`/`PreferredTarget`) persist on `CrossDriveBackup` and survive the runner's status
|
||
> writes (`withTier2Prefs`). Always visible incl. single-SSD + non-HDD (PBS-context) apps.
|
||
> - Part B (storage OS/data split spike) = build-nothing; findings → `felhom-agent/REPORT-storage-split-spike.md`.
|
||
> - Live-validated on guest 9201; build/deploy = golden bootstrap (`/etc/felhom-controller-image` + restart `felhom-controller-bootstrap.service`).
|
||
>
|
||
> **2026-06-13 — v0.56.0 Phase 4: FileBrowser scoping + UI polish (SLICE COMPLETE):**
|
||
> - 4A: FileBrowser bind scoped to `<drive>/appdata` (recovery units + Tier 2 copies under `backups/`
|
||
> NOT mounted → customer can't browse/delete the restore source). 4B: deploy storage step states
|
||
> files-on-drive / DB-on-fast-SSD. 4C: `buildStorageBars` stable sort + purpose description on the
|
||
> monitoring list (user-data drives only; agent local/local-lvm/pbs live on the storage page, not here).
|
||
> - Live-validated (9201): FileBrowser mount `/mnt/felhom-usb/appdata -> /srv/felhom-usb` (backups hidden);
|
||
> deploy + monitoring text rendered. **All 5 phases (1, 2, 2b, 3, 4) shipped + live-validated, v0.52→v0.56.**
|
||
>
|
||
> **2026-06-13 — v0.55.0 Phase 3: auto off-drive Tier 2 (rootfs-headroom guard):**
|
||
> - `internal/backup/tier2.go`: rsync `-a --delete` of each HDD app's recovery unit + appdata → a
|
||
> DIFFERENT physical disk (`<target>/backups/secondary/<app>/`). Auto target: prefer another registered
|
||
> drive (off-disk via `system.SamePhysicalDevice`), else internal SSD for SMALL units only.
|
||
> - **Rootfs-headroom guard** (`tier2FitsHeadroom`, unit-tested): SSD = ~8G guest rootfs, so REFUSE
|
||
> unless the unit fits leaving reserve = max(2G, 20%) free; honest "needs 2nd HDD" status when nothing
|
||
> fits — never fills the rootfs. Status via surviving `settings.CrossDriveBackup`; "2. mentés" UI card
|
||
> now populated (`buildAppBackupRows`). Daily `tier2-backup` 03:30 + `POST /api/backup/tier2`.
|
||
> - **Live-validated (9201):** happy path (RomM → SSD, off felhom-usb, 77KB, "[SSD: DB/config only]");
|
||
> refuse path (1G userdata dummy → REFUSED with honest msg, rootfs not filled); UI card shows
|
||
> "Sikeres → belső SSD (csak DB/konfiguráció)". Demo cleaned.
|
||
> - Next: Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).
|
||
>
|
||
> **2026-06-13 — v0.53.0/v0.53.1 Phase 2: per-app recovery unit (capture side, SECRET-FREE):**
|
||
> - Each app's `backups/primary/<app>/` becomes a self-contained recovery unit: `compose/`
|
||
> (docker-compose.yml + .felhom.yml + **secret-stripped** app.yaml) + db-dumps/ + volume-dumps/ +
|
||
> `manifest.json` (image pins, secret env-var NAMES, data_key names, checksums, secret_source note).
|
||
> - **Secret-free by design.** Decided after reading the ACTUAL hub code: hub is zero-knowledge (no app
|
||
> secrets); app.yaml + key live on the guest rootfs → in the PBS whole-guest snapshot. So the unit
|
||
> stores no secret/data-key/image; restore recovers secrets from the guest's app.yaml (live/PBS),
|
||
> regenerates nothing. `data_key` (DeployField.DataKey; AdventureLog SECRET_KEY marked) = fail-closed
|
||
> restore annotation only.
|
||
> - Capture needs no decryption (non-secret env is plaintext; excludes secret-named + encrypted keys).
|
||
> Wired into RunDBDumps AND the periodic RefreshCache (idempotent checksum-skip → no USB thrash).
|
||
> - **Deploy mechanism resolved:** controller in guest 9201 is golden/bootstrap-managed —
|
||
> `felhom-controller-bootstrap.service` docker-runs the tag from `/etc/felhom-controller-image`
|
||
> (gitea anon-pull). Deploy = build+push → anon-pull → update tag file → restart the service.
|
||
> - **Live-validated (9201):** RomM unit captured (images=3, secrets=3, data_keys=0), secret-leak grep
|
||
> = NO_LEAK.
|
||
> - **v0.54.0 Phase 2b (restore-from-unit + fail-closed gate):** `RestoreFromRecoveryUnit` recreates an
|
||
> app from its unit + secrets recovered from the GUEST's live app.yaml (`RecoverStackSecrets`,
|
||
> `stacks.RedeployFromEnv`), regenerating nothing. `reconcileRestoreSecrets` (pure, unit-tested) is the
|
||
> fail-closed gate: missing/empty data-key → REFUSE (needs PBS whole-guest restore); missing resettable
|
||
> secret → warn+proceed. Wired into `/backup/restore`. Gate + orchestration + data_key parsing
|
||
> unit/integration-tested; deployed v0.54.0 healthy.
|
||
> - **LIVE-validated (9201, AdventureLog):** unit manifest `data_key_env_vars:[SECRET_KEY]`
|
||
> (catalog→manifest live); with SECRET_KEY made unrecoverable, `POST /backup/restore` REFUSED with the
|
||
> exact fail-closed message BEFORE any compose-up. Demo has NO dashboard password → API open (auth+CSRF
|
||
> skipped), driven via public URL. NOTE: full deploy-with-data→restore e2e blocked because AdventureLog
|
||
> images don't fit the 8G guest rootfs ("no space left") — that's the Phase 3 rootfs-headroom concern
|
||
> seen live. Demo left clean (AdventureLog reverted to not-deployed).
|
||
> - Next: Phase 3 (Tier 2 auto off-drive, rootfs-headroom guard), Phase 4 (FileBrowser + UI).
|
||
>
|
||
> **2026-06-13 — v0.52.0 Phase 1 GATE: deploy-side double-nest fix (catalog) + path-agreement test:**
|
||
> - The `felhom-data` double-nest lived in the **app-catalog compose templates**
|
||
> (`${HDD_PATH}/felhom-data/appdata/<app>`), not in `deploy.go`. On a Model-A in-guest drive the mount
|
||
> already IS the `felhom-data` namespace, so it double-nested on disk while the v0.51.0 backup helpers
|
||
> resolved single-nested → divergence. Fixed all four HDD templates (romm, nextcloud, immich,
|
||
> paperless-ngx) → `${HDD_PATH}/appdata/<app>`.
|
||
> - New `internal/stacks/hddpath_agreement_test.go` locks deploy-resolver (`ParseComposeHDDMounts`) ==
|
||
> backup helper (`AppDataDir(NamespaceRoot(.,true))`). No controller runtime change → no image rebuild
|
||
> (deployed stays 0.51.0, functionally current; golden not rebaked for a no-op).
|
||
> - **Live (guest 9201):** git-sync auto-delivered the fix to all four stack files; RomM migrated
|
||
> (stop→move→verify→redeploy) from `/mnt/felhom-usb/felhom-data/appdata/romm` →
|
||
> `/mnt/felhom-usb/appdata/romm`, healthy + HTTP 200, no data loss, old namespace empty. **GATE PASSED.**
|
||
> - Next: Phase 2 (per-app recovery unit), Phase 3 (auto-enabled off-drive Tier 2 w/ rootfs-headroom
|
||
> guard), Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).
|
||
>
|
||
> **2026-06-12 — storage UX polish (v0.45.0), pairs with felhom-agent v0.24.0:**
|
||
> - **Agent eject role-gate (Part A, felhom-agent v0.24.0):** `POST /disks/eject` now refuses to
|
||
> unmount system/backup storage *at the agent* (fail-safe to protected on ambiguity) — the UI hiding
|
||
> the button was never the control. Validated live on guest 9201 (eject `/var/lib/vz` → 403, no unmount).
|
||
> - **Controller (Part B, v0.45.0):** B1 deterministic `/api/disks` order (user-data→system→backup,
|
||
> alpha within); B2 init wizard excludes mounted drives; B3 **Regisztrálás** primary action for a
|
||
> mounted-but-unregistered user-data drive (`POST /api/storage/register`); B4 per-card purpose
|
||
> descriptions + app-backing tags + tiering note (`local` & `local-lvm` both kept); B5 eject already
|
||
> names affected apps. All validated live on guest 9201.
|
||
> - **Golden REBAKED** to controller 0.45.0 (`/root/build-golden.sh`; gitea allows anon pull, no creds);
|
||
> archive `local:backup/vzdump-lxc-9100-2026_06_12-09_53_03.tar.zst`, build guest 9100 purged.
|
||
|
||
---
|
||
|
||
## About Viktor (project owner)
|
||
|
||
- Works at Deutsche Telekom (Budapest), building Felhom.eu as a side business
|
||
- Felhom.eu: managed home-server service for Hungarian households
|
||
- Technical but prefers pragmatic solutions over over-engineering
|
||
- Runs all infrastructure on Gitea (gitea.dooplex.hu), k3s cluster for management
|
||
- Customer deployments use Docker Compose (not Kubernetes) for simplicity
|
||
|
||
### felhom-controller (this repo)
|
||
- **Version:** v0.16.1
|
||
- **Phase 1:** ✅ COMPLETE — Stack Manager + Deploy Flow
|
||
- **Phase 2:** ✅ COMPLETE — Monitoring & Health (scheduler, CPU/temp, healthchecks.io pings)
|
||
- **Phase 3:** ✅ COMPLETE — Backups (DB dumps, restic integration, manual trigger, **dedicated backup page**)
|
||
- **Phase 4:** ✅ COMPLETE — Monitoring Page with Metrics Store (SQLite, Chart.js, system + container metrics)
|
||
- **Phase 5:** ✅ COMPLETE — Authentication, Persistence & Settings Page (settings.json, password change, session management)
|
||
- **Phase 6:** ✅ COMPLETE — Monitoring Warnings, Dashboard Alerts & Notification System
|
||
- **Phase 7:** ✅ COMPLETE — Storage Overview, Per-App Backup Toggles & Limited Restore
|
||
- **Phase A:** ✅ COMPLETE — Storage Paths Foundation (registry, auto-discovery, per-app HDD_PATH, deploy dropdown, health monitoring)
|
||
- **Phase B:** ✅ COMPLETE — Storage Management UI Polish & Health Severity Fix (flash messages, label editing, app details, FS info, deploy free space, backup context)
|
||
- **Phase C:** ✅ COMPLETE — Storage Init Wizard, Data Migration & Startup Fix (disk scan/format/mount wizard, rsync-based migration, startup pings)
|
||
- **v0.11.1 bugfix:** ✅ COMPLETE — Storage Scan: system disk detection via host fstab + blkid UUID resolution; FSType enrichment via `blkid -o export`
|
||
- **v0.11.2 bugfix:** ✅ COMPLETE — /host-dev mount for block device access; `HostDevicePath()` helper; all format/scan/safety ops use /host-dev
|
||
- **v0.11.3 bugfix:** ✅ COMPLETE — Added `fdisk` package to Dockerfile (provides `sfdisk`; not in `util-linux` on Debian bookworm)
|
||
- **v0.11.4 bugfix:** ✅ COMPLETE — FormatAndMount: fixed sfdisk (wipefs+force+`,,`), mount (explicit device path), mount propagation (rshared), ASCII label, smart partition skip, findmnt verification
|
||
- **v0.11.6:** ✅ COMPLETE — FileBrowser auto-mount sync (`syncFileBrowserMounts()`) + 3 UI fixes (badge color, progress bar, button text)
|
||
- **v0.11.7:** ✅ COMPLETE — Stale data cleanup + FileBrowser sync after migration + deploy page title fix
|
||
- **v0.11.8:** ✅ COMPLETE — Per-App Cross-Drive Backup (3-2-1 rule): rsync/restic to secondary drive, deploy page UI, backup page summary, scheduler jobs, API endpoints
|
||
- **v0.11.9:** ✅ COMPLETE — UI Polish Fixes: spacing, tooltip on "Módszer", status dot instead of disabled checkbox, progressive disclosure, emoji cleanup
|
||
- **First app deployed:** Paperless-ngx on demo-felhom.eu (2026-02-13)
|
||
- **Running on:** demo-felhom (N100 mini PC) at 192.168.0.162:8080, felhotest (Proxmox VM) at router.abonet.hu:33022
|
||
- **All Phase 1-5 features working:** deploy, start/stop/restart/update, logs, health-aware states, auth, monitoring, backups, backup detail page, system monitoring page, settings page
|
||
|
||
## Architecture decisions
|
||
|
||
| Decision | Rationale |
|
||
|----------|-----------|
|
||
| Go stdlib for web (no Gin/Echo) | Minimal dependencies, single binary, easy to embed templates |
|
||
| Templates as go:embed HTML/CSS files | Zero runtime file dependencies (compiled into binary), but each template is a separate editable file |
|
||
| Docker Compose for customers (not k8s) | Simpler troubleshooting, customers don't need k8s knowledge |
|
||
| k3s for management infra only | Viktor's own services (gitea, monitoring, website) run on k3s |
|
||
| Cloudflare Tunnel for remote access | No port forwarding needed, works behind any NAT |
|
||
| app.yaml per stack | Separates deploy config from compose files, survives git pulls |
|
||
| Password fields require explicit input | Prevents accidental empty-password deployments |
|
||
| Health-aware state from Docker Status field | Docker's State says "running" even for unhealthy containers |
|
||
| Memory limits via deploy.resources.limits | Prevents runaway containers; ~50% headroom over expected usage |
|
||
| System info from /proc/meminfo + statfs | No external dependencies, cheap to read on each page load |
|
||
| mem_request vs mem_limit (K8s-inspired) | Requests = expected usage (hard block), limits = peak (overcommit OK) |
|
||
| 384MB reserved for system | Prevents deploying apps that would starve the OS/controller |
|
||
| Logo SVG embedded as Go constant | Same approach as CSS/HTML — zero external file deps |
|
||
| Git sync via os/exec git CLI | No Go git library needed, git is in the container image |
|
||
| SHA-256 for content comparison | Only copy changed files, avoid unnecessary disk writes |
|
||
| 30s debounce on manual sync | Prevents spamming the git server |
|
||
| Orphan = deployed but not in catalog | Safe lifecycle: remove from catalog → mark orphaned → user deletes via UI |
|
||
| FileBrowser as infra (not catalog) | Needed even after apps deleted (user browses HDD data); deployed by setup script |
|
||
| Protected HDD paths | Safety net: never delete top-level HDD dirs (media, storage, Dokumentumok, appdata) |
|
||
| Central scheduler (not ad-hoc goroutines) | Single place to register/monitor all periodic tasks, graceful shutdown, skip-if-running |
|
||
| CPU sampling via background goroutine | /proc/stat delta needs two readings — collector runs every 5s, GetInfo() reads cached value |
|
||
| Temperature from /host/sys (Docker mount) | Container can't read host /sys directly — mount /sys:/host/sys:ro, try /host/sys first |
|
||
| Restic password auto-generated | No manual setup needed — generated on first backup run, stored in named volume |
|
||
| DB discovery via docker inspect | No config needed — discovers postgres/mariadb containers by image name + env vars |
|
||
| Backup orchestrator with running flag | Prevents concurrent backups, supports both scheduled and manual trigger |
|
||
| modernc.org/sqlite (pure Go) | No CGO/gcc needed in Docker build stage — keeps `CGO_ENABLED=0` static binary |
|
||
| AlertManager state-based refresh | Alerts regenerated every 5min from health report — no persistent storage needed, always reflects current state |
|
||
| Notification relay via hub | Controller → hub → Resend → email. Hub acts as central relay: knows customer email, handles Resend API. Controller only needs hub URL + API key |
|
||
| In-memory notification cooldowns | Per-event-type cooldown map (default 6h). Lost on restart = acceptable (better to re-notify than miss). No persistence needed |
|
||
| Health status change detection | Only notify on degradation (ok→warn, ok→fail, warn→fail). Avoids spam on flapping. First run records baseline, doesn't notify |
|
||
| Resend HTTP API (no SMTP) | Direct POST to api.resend.com — same pattern as website contact-mailer. Simpler than SMTP setup, good deliverability |
|
||
| Preferences sync on save + startup | Controller pushes prefs to hub (not pull). Startup sync handles hub DB rebuild. Local save always succeeds even if sync fails |
|
||
| Chart.js embedded locally | Customer hardware may not have internet — CDN not reliable for offline environments |
|
||
| StackDataProvider interface | backup package needs stack data but can't import stacks (circular). Interface in backup, thin adapter in main.go |
|
||
| Password sync to hub via report | Restic password in Docker named volume on SSD. Hub sync provides redundancy for disaster recovery |
|
||
| App backup via HDD mounts only | Docker volumes at /var/lib/docker/volumes/ not mounted in controller. HDD data is the important user data; DB in volumes covered by nightly dump |
|
||
| Restore uses running mutex | Prevents concurrent backup+restore on same restic repo. Reuses existing `m.running` flag |
|
||
| Storage paths registry in settings.json | Multi-storage support: each app's HDD_PATH from app.yaml is authoritative. Auto-discovery on startup avoids manual config. Registry enables UI management + health monitoring per path |
|
||
| /mnt:/mnt:rw mount in controller | Replaces per-path HDD_PATH mount. Enables multi-storage + restore writes. All customer HDD mounts are under /mnt/ by convention |
|
||
| Per-app HDD_PATH resolution (app.yaml > global) | App's own env HDD_PATH is Priority 1, registered storage paths as fallback. Eliminates dependency on global controller.yaml hdd_path |
|
||
| Mount-point detection via syscall.Stat_t.Dev | Compares device ID of path vs parent dir — reliable check that path is on separate filesystem. Prevents data writes to SSD |
|
||
| Health severity: mount-point = warning | Non-mount-point is informational, not a service failure. FAIL reserved for genuinely broken things. Avoids false alarms on demo/test environments |
|
||
| FS info via findmnt + sysfs | `findmnt -n -o SOURCE,FSTYPE --target <path>` for filesystem type/device. `/sys/block/<dev>/device/model` for disk model. Best-effort, returns nil on failure |
|
||
| Query param flash messages | Stateless, no session store needed. Consistent with backup page pattern. `?storage_msg=success&storage_detail=...` |
|
||
| StorageLabels map on stacks page | Separate map passed to template (not modifying Stack struct). Built from deployed apps' HDD_PATH → registered path label lookup |
|
||
| Metrics downsampling via SQL | Bucket-based AVG in GROUP BY keeps Chart.js responsive with up to 30 days of data |
|
||
| 60s metrics collection interval | Good balance of resolution vs. storage — ~44K rows/month for system metrics |
|
||
| /etc/os-release mounted read-only | Container can't read host OS info directly — mount to /host/etc/os-release:ro |
|
||
|
||
## Key file locations on demo-felhom
|
||
|
||
```
|
||
/opt/docker/felhom-controller/ # Controller compose + config
|
||
├── controller.yaml # Customer config (domain, auth, paths)
|
||
├── docker-compose.yml # Controller's own compose
|
||
└── data/ # Controller persistent data (named volume)
|
||
|
||
/opt/docker/stacks/ # All app stacks
|
||
├── traefik/ # Reverse proxy (protected)
|
||
├── cloudflared/ # Tunnel (protected)
|
||
├── paperless-ngx/ # First deployed app ✅
|
||
│ ├── docker-compose.yml
|
||
│ ├── .felhom.yml # App metadata
|
||
│ └── app.yaml # Deploy config (env vars, locked fields)
|
||
└── whoami/ # Test stack (not deployed)
|
||
|
||
/mnt/hdd_placeholder/storage/ # HDD storage for apps
|
||
└── paperless/
|
||
├── consume/ # Drop files here for OCR
|
||
├── media/ # Processed documents
|
||
└── export/ # Backup exports
|
||
```
|
||
|
||
## Related repositories and their state
|
||
|
||
| Repository | Status | Notes |
|
||
|------------|--------|-------|
|
||
| felhom-controller | Active | This repo. Controller code + deploy scripts |
|
||
| app-catalog-felhom.eu | Active | 10 app templates, all with .felhom.yml metadata + memory limits |
|
||
| felhom.eu | Active | Website + hub/ subfolder (felhom-hub service) + k8s manifests |
|
||
| homelab-manifests | Stable | k3s cluster running (dooplex.hu services) |
|
||
| misc-scripts | Utility | collect-repo.sh, backup helpers |
|
||
|
||
## Gotchas & lessons learned
|
||
|
||
- `docker compose restart` ≠ `docker compose up -d` — restart doesn't pick up new images
|
||
- Go maps have random iteration order — always sort slices before displaying
|
||
- Docker `.State`="running" doesn't mean healthy — check `.Status` for "(health: starting)" / "(unhealthy)"
|
||
- Paperless-ngx needs `PAPERLESS_OCR_LANGUAGES` (plural) to install language packs, `PAPERLESS_OCR_LANGUAGE` (singular) to select
|
||
- In-memory Deployed flag must be set BEFORE `docker compose up -d` (not after) — compose can take 30-60s for image pulls, during which the UI would show a stale "Telepítés" button
|
||
- Cloudflare Tunnel handles *.demo-felhom.eu → Traefik handles Host()-based routing to containers
|
||
- BIOS "AC Power Recovery" must be enabled on N100 for auto-restart after power outage
|
||
- `docker compose up -d` returns exit 0 even when containers immediately crash-loop — need post-start status check to detect this
|
||
- When logging env vars for debugging, only log keys (not values) to avoid leaking secrets in log files
|
||
- Mealie image (`ghcr.io/mealie-recipes/mealie`) doesn't include wget/curl — use Python TCP socket check for healthcheck
|
||
- Mealie DB migrations on first start take ~40s (alembic) — use `start_period: 60s` to avoid premature unhealthy status
|
||
- Alpine-based images (filebrowser, vaultwarden) have wget via BusyBox — healthchecks with `wget --spider` work fine
|
||
- Deploy `sed` command to update image version must target only the `image:` line — naive `sed 's|name:OLD|name:NEW|'` also matches the service name line (e.g., `felhom-controller:` → `felhom-controller:0.2.12`), breaking YAML. Use `sudo sed -i 's|image:.*felhom-controller:[^ ]*|image: ...felhom-controller:NEW|'` or similar scoped pattern
|
||
- Hungarian quotation marks `„"` in YAML: `„` (U+201E) is safe inside YAML double-quoted strings, but the closing `"` must NOT be ASCII `"` (0x22) — it terminates the YAML string. Use `\"` escape or Unicode `"` (U+201D). This caused a silent parse failure for the entire `.felhom.yml` file
|
||
- Never silently swallow parse errors — always log them. Silent failures make debugging impossible (took a dedicated debug session to find a simple quoting issue) |