felhom.eu/documentation/controller/deploy-and-stack-lifecycle.md

# Controller: deploy & stack lifecycle

> Source of truth: felhom-controller `internal/stacks/`, `internal/infra`, `internal/sync` at v0.59.0.

This document describes how the in-guest controller represents application stacks, deploys
them, runs their lifecycle (update/stop/start/restart/remove/delete), and brings up the base
infrastructure (traefik / cloudflared / filebrowser) that everything else depends on. Every
claim below is verified against the current source.

## 1. The stack model

A *stack* is one customer app, represented on disk as a directory under
`/opt/docker/stacks/<name>/` (config key `paths.stacks_dir`, default `/opt/docker/stacks`)
containing:

- `docker-compose.yml` — the compose template (synced from the catalog, see §7).
- `.felhom.yml` — app metadata (`Metadata`): display name, deploy fields, resource
  requests/limits, optional config, healthcheck spec, integrations.
- `app.yaml` — the per-app deployment record (`AppConfig`), written **only after** the app
  is first deployed.

In memory each stack is a `Stack` struct (`internal/stacks/manager.go:62`):

| Field | Meaning |
|---|---|
| `State` (`ContainerState`) | aggregated container state |
| `Deployed` | has an `app.yaml` with `deployed: true` |
| `Deploying` | a `compose up` is in progress (image pull, etc.) |
| `Protected` | a system stack that must never be stopped/removed (§4) |
| `Orphaned` | deployed but no longer present in the synced catalog |
| `DeployError` | last async deploy error, surfaced to the UI |
| `HealthProbe` | latest controller-side probe result (§6) |

`ContainerState` values (`manager.go:23`): `running`, `starting` (`health: starting`),
`unhealthy`, `stopped`, `restarting`, `exited`, `paused`, `unknown`, `not_deployed`,
`deploying`, `orphaned`. The effective state is derived by `resolveContainerState`
(combines Docker `State` + the health hint in `Status`) and then `aggregateState` across
all of a stack's containers, with priority `unhealthy > starting > restarting > all-running
> stopped` (`manager.go:437`, `:468`).

`ScanStacks` (`manager.go:250`) discovers stacks by walking `stacks_dir`, reading metadata
and `app.yaml`, marking `Protected` via `cfg.IsProtectedStack`, and computing `Orphaned`
against the synced catalog cache (`getCatalogTemplateSlugs`, `manager.go:1078`). While an
async deploy is in flight it deliberately does **not** overwrite `Deployed`/`AppConfig` —
the deploy goroutine owns those (H3 fix, `manager.go:293`).

### `AppConfig` (`app.yaml`)

```go
type AppConfig struct {
    Deployed     bool              `yaml:"deployed"`
    DeployedAt   string            `yaml:"deployed_at"`
    Env          map[string]string `yaml:"env"`
    LockedFields []string          `yaml:"locked_fields"`
}
```
(`deploy.go:98`). `Env` holds the resolved deploy-field values; sensitive ones are stored
**encrypted** (§3). `LockedFields` lists env vars marked `locked_after_deploy` that
`UpdateStackConfig` refuses to change (`deploy.go:434`).

## 2. Deploy flow

Entry point: `DeployStack(req)` (`deploy.go:121`), driven by the deploy form ("Telepítés").

1. **Atomic check-and-set of `Deploying` (H1).** Under the manager lock, the stack must
   exist, must not already be `Deploying`, and must not already be `Deployed`; then
   `Deploying` is set true (`deploy.go:122`). This single critical section prevents two
   concurrent deploys of the same app. Any later validation failure clears the flag via
   `clearDeploying()`.

2. **Memory gate.** `system.GetMemoryMB()` vs the app's `mem_request` and the configured
   `reserved_memory_mb`. A hard block (real used + new request > usable) aborts with a
   Hungarian error; a soft over-commit case returns a non-fatal warning string
   (`deploy.go:159`).

3. **Field resolution.** For each `DeployField`: `domain` is auto-filled from
   `customer.domain`; `subdomain` is validated (DNS-safe, not reserved, not already in
   use); `secret` uses the value previewed to the user or generates one; `password` is
   **required** (never silently generated — the user must know it); `path` must exist on
   the host. Required-but-empty fails (`deploy.go:206`–`292`).

4. **`app.yaml` is persisted with `deployed: false` first (CTRL-T2-1).** The in-memory
   `AppConfig` carries `Deployed: true` (for UX), but a clone with `Deployed: false` is what
   `SaveAppConfig` writes to disk at this point (`deploy.go:302`–`314`). The durable record
   is intentionally *not* marked deployed yet.

5. **In-memory state flips to deployed** (`s.Deployed = true`, `s.AppConfig = appCfg`),
   so the UI immediately stops showing a stale "Telepítés" button during the pull
   (`deploy.go:331`).

6. **Async compose.** `go m.runComposeDeploy(...)` runs `docker compose up -d` off the
   request path, so the HTTP handler returns at once and the UI polls for progress (an
   image pull can take 30–60s). The compose command injects the resolved env plus `DOMAIN`
   (`deploy.go:338`, `composeExecWithEnv` `manager.go:505`).

### CTRL-T2-1 crash-safety (the key current behaviour)

The on-disk `app.yaml` records `deployed: true` **only after** `docker compose up -d`
actually succeeds. `runComposeDeploy` (`deploy.go:346`):

- On compose **failure**: reverts in-memory state (`Deployed=false`, `Deploying=false`,
  records `DeployError`, `AppConfig=nil`) and re-saves the reverted `AppConfig`
  (still `deployed:false`) to disk (`deploy.go:350`).
- On compose **success**: re-runs `SaveAppConfig` with `appCfg.Deployed == true`, flipping
  the durable record to deployed (`deploy.go:371`). If *that* save fails, it reverts in
  memory so the customer can cleanly redeploy rather than be stuck half-recorded
  (`deploy.go:375`).
- Only then is `Deploying` cleared and `RefreshStatus()` run.

Why it matters: a crash or power loss **during the image-pull window** leaves `app.yaml`
at `deployed: false`. On restart, `ScanStacks` therefore sees the app as *not deployed*
(no ghost-deployed stack with no containers), and `DeployStack` will cleanly redeploy
instead of refusing with "already deployed".

The UI's three-step progress (form submit → "pulling/deploying" via the `deploying`
state and `DeployError` → settled `running`/`unhealthy`) is driven entirely by this
in-memory state, which the front-end polls.

## 3. `SaveAppConfig` and secret encryption (fail-closed, H10)

`SaveAppConfig(stackDir, cfg, encKey, sensitiveVars)` (`deploy.go:669`) clones the env and,
for each var named in `sensitiveVars` (the `secret`/`password` deploy fields, computed by
`SensitiveEnvVars`, `deploy.go:741`), encrypts the value with the AES-256 key
(`crypto.Encrypt`) unless it is empty or already encrypted.

**Fail-closed (H10, v0.59.0):** if encryption of a sensitive value errors, `SaveAppConfig`
**returns the error and writes nothing** — it never falls through to a plaintext write
(`deploy.go:683`–`691`). Earlier code logged a WARN and persisted plaintext, which leaked
the secret to disk; that path is gone. Callers (`DeployStack`, `UpdateStackConfig`, etc.)
already propagate this error, so the deploy fails cleanly with no secret on disk.

The write itself is atomic: write to `app.yaml.tmp` then `rename` (H04, `deploy.go:711`),
mode `0600`. Decryption for compose happens lazily via `LoadAppConfigDecrypted` /
`stackEnv` (`deploy.go:727`, `manager.go:809`), which injects `DOMAIN` plus the decrypted
env into the compose process environment. A one-time startup `MigrateEncryption`
(`manager.go:128`) re-saves any deployed app still holding plaintext sensitive values.

## 4. Lifecycle: update / stop / start / restart / remove / delete

All operations resolve the stack dir, build env via `stackEnv` (decrypted), and shell out
to compose. Notably **start/restart use `up -d`, not bare `restart`**, so env changes and
template changes (new images, healthchecks) are picked up (`manager.go:709`, `:643`).

- **Start** (`StartStack`, `manager.go:643`): `compose up -d`; clears the stale health
  probe so the next probe runs fresh.
- **Stop** (`StopStack`, `manager.go:682`): refuses protected stacks, then `compose down`.
- **Restart** (`RestartStack`, `manager.go:709`): `compose up -d`; clears health probe.
- **Update** (`UpdateStack`, `manager.go:746`): `compose pull` then
  `compose up -d --remove-orphans`.
- **Config update** (`UpdateStackConfig`, `deploy.go:405`): rejects locked fields, re-saves
  `app.yaml` (encrypting secrets), then `up -d` with the decrypted env.
- **Remove** (`RemoveStack`, `delete.go:283`): for a *deployed* (non-orphaned) stack —
  `compose down --volumes` (keeps images for redeploy), optional HDD-data and backup-path
  cleanup, then deletes **only** `app.yaml`, reverting the stack to "not deployed". Compose
  template files are preserved so the user can redeploy.
- **Delete** (`DeleteStack`, `delete.go:80`): for an *orphaned* stack only —
  `compose down --rmi local --volumes`, optional HDD-data removal, then removes the whole
  stack directory.

### Protected-stack enforcement (server-side, defence in depth)

The protected set is configured under `stacks.protected` (default
`traefik`, `cloudflared`, `felhom-controller`, `filebrowser` — see
`configs/controller.yaml.example:55` and the setup writer `internal/setup/handlers.go:476`).
`config.IsProtectedStack` matches case-insensitively (`config.go:347`). Enforcement exists
at three layers:

- **Router/API:** `actionStack` blocks any action other than `restart` on a protected
  stack with HTTP 403 (`internal/api/router.go:412`).
- **Manager:** `StopStack` (`manager.go:683`), `RemoveStack` (`delete.go:289`) and
  `DeleteStack` (`delete.go:86`) each independently refuse protected stacks, so the guard
  holds even if a caller bypasses the router.
- The quiesce loop's `RunningAppStacks` (`manager.go:232`) also excludes protected stacks,
  so the controller never stops its own tunnel/proxy or itself for a backup.

### Deploying-guard on remove/delete (H2)

Both `RemoveStack` (`delete.go:309`) and `DeleteStack` (`delete.go:106`) refuse while the
stack is `Deploying`, and both also require the stack to be stopped (not
running/starting/restarting). HDD-path deletion is gated by `ProtectedHDDPaths`
(`delete.go:58`), which refuses to wipe top-level namespace directories (`appdata/`,
`backups/`, `media/`, etc.), and `ParseComposeHDDMounts` cleans paths before the prefix
check to block traversal (C10, `delete.go:555`).

## 5. Base-infra bring-up: `EnsureBaseStack`

`EnsureBaseStack` (`internal/stacks/infra.go:27`) renders and deploys the routing/access
infrastructure. Properties (verified):

- **Single-flight:** guarded by `infraMu.TryLock()` (`infra.go:28`). It is fired both at
  first boot and on every `system-health` tick (self-heal); a second concurrent invocation
  returns immediately rather than racing a `compose up` on the same dir.
- **Idempotent:** each component is skipped when its container is already running
  (`containerRunning`, `infra.go:256`), so the healthy-state re-run is a cheap trio of
  `docker inspect` calls.
- **Non-fatal by contract:** per-component failures are collected into a joined error for
  the caller to *log* — it must never crash the controller (`infra.go:67`).

Deploy order is load-bearing (the composes declare `traefik-public` as
`external: true`, so it must exist first):

1. `ensureTraefikNetwork` — creates the external `traefik-public` docker network if absent,
   tolerating a create/inspect race (`infra.go:211`). A failure here is fatal to the run
   (every stack `up` would fail without it).
2. `ensureTraefik` — prepares `dynamic/`, `certs/`, a `0600` `acme.json`, renders the
   traefik config from `customer.email` + `infrastructure.cf_api_token`, writes it, and
   `compose up -d` (`infra.go:73`).
3. `wireController` — writes the file-provider route `Host(felhom.<domain>) →
   http://felhom-controller:8080` and joins the `felhom-controller` container to
   `traefik-public`. Both steps are idempotent (route rewritten only when content changed,
   so the traefik file watcher doesn't reload every tick; network-connect skipped when
   already attached). A missing domain is a logged no-op, not an error (`infra.go:160`).
4. `ensureCloudflared` — **only when `infrastructure.cf_tunnel_token` is set**; a LAN-only
   node legitimately runs without it, and `monitor.EffectiveProtected` mirrors this
   condition so such a node doesn't report cloudflared as a perpetually-missing protected
   container (`infra.go:55`, `monitor/healthcheck.go:252`).
5. `ensureFileBrowser` — **preserves an existing compose.** If
   `filebrowser/docker-compose.yml` already exists, it `compose up -d` *without
   regenerating*, so the storage mounts that `web.SyncFileBrowserMounts` manages are kept
   intact. Only on first provision does it render the initial compose + `config.yaml` with
   no mounts (`infra.go:122`).

## 6. Health probes

`RunHealthProbes` (`internal/stacks/healthprobe.go:23`) is a scheduler-driven, controller-side
probe (independent of Docker's own healthcheck). It only probes stacks in `running`/`unhealthy`
state that declare a `healthcheck` in metadata and whose interval has elapsed. The interval is
adaptive: a fast **10s** retry while unhealthy, otherwise the configured interval (default 5m);
when `HealthProbe` is nil (just started) it probes immediately (`healthprobe.go:41`).

Targets are collected under a read-lock, probed **concurrently** with the lock released
(`healthprobe.go:91`), and results applied under the write-lock. Check types: `tcp`
(dial), `http` (any response = healthy), and `api` (validate expected status / body
substring) — `runSingleCheck`, `probeTCP`, `probeHTTP` (`healthprobe.go:169`+). A failed
probe overrides a Docker-`running` stack to `unhealthy`; a passing probe clears that
override back to `running`. `refreshStatusLocked` re-applies the last probe result so a
status refresh never resurrects a stale healthy state (`manager.go:418`).

## 7. How stacks get their templates (catalog git-sync)

The compose templates and metadata come from the app catalog via `internal/sync`
(`Syncer`, `sync.go:22`). On a configurable interval (`git.sync_interval`, default 15m,
with an initial sync at startup) it shallow-clones/pulls `git.repo_url` (branch
`git.branch`) into `<data_dir>/catalog-cache` via `git fetch --depth 1` +
`reset --hard origin/<branch>` (`sync.go:247`). It then copies **only** `docker-compose.yml`
and `.felhom.yml` from each `templates/<app>/` into `stacks_dir/<app>/`, hashing content
to skip unchanged files and **never** overwriting `app.yaml` (`copyTemplates`,
`sync.go:311`; `copyIfChanged`, `sync.go:395`).

After a sync that changed anything, it triggers a rescan (`ScanStacks`) and, for *updated*
stacks, a post-sync hook that runs `InjectMissingFields` — auto-generating values for any
new `secret`/`domain`/`subdomain` deploy fields that aren't yet in a deployed app's
`app.yaml` (`sync.go:207`, `deploy.go:801`). `TriggerSync` (manual) debounces to one run
per 30s. Sync is disabled (manual mode) when `git.repo_url` is empty (`sync.go:79`). The
catalog cache also feeds orphan detection (`getCatalogTemplateSlugs`): a deployed stack
whose slug is no longer in the cache is flagged `Orphaned`.