felhom.eu/documentation/architecture/02-controller-module-map.md

# Felhom Controller Architecture — Part 2: Controller Module Map

> **How to read this document.** Two kinds of statement appear, and where this document marks them it
> marks them like this — the same wording as `07-backup-architecture.md:11-17`, carried here on
> 2026-08-22 (R-376) so a reader meets one convention and not eight:
>
> - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet.
> - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation.
>
> **Statements in this document are NOT yet all marked.** Marking them wholesale is a large judgement
> exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376).
> **An unmarked statement therefore means "not yet classified", never "observed".** That ambiguity is
> exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat
> unmarked beside a marked `[FACT]`, and was read as an observation and reported as a defect.



> **EXECUTED (slice 8C, 2026-06-10 — controller v0.37.0).** This map's target state is now realized:
> the disk-execution subsystem (`storage/*`, restic, cross-drive, drive-restore, `disk_layout`,
> `local_infra`, `infra_backup`, `setup/scanner`, `monitor/watchdog`+`pinger`, the storage UI) is
> **deleted** (~12.3k LOC); `backup.Manager` is **split to app-data only**; disk management is
> **rewired to the host agent's local API** (`web/agent_disk_handlers.go` → agent `/disks`); and the
> container is **de-privileged** (no `privileged`, `/dev`, `/etc/fstab`, rshared). The in-guest
> controller is now **Docker-only with no disk/Proxmox privileges**, as designed. See doc 03 §6/§9.

**Status:** audit (keep / port / delete / modify / add), grounded in the v0.33 source.
**Subject:** the v0.33 controller in `felhom-controller/controller/` (110 `.go` files,
~40 K LOC) audited against [01-topology-and-trust.md](01-topology-and-trust.md) and
[../proxmox-platform.md](../proxmox-platform.md).

> This is a **planning map, not the port.** No controller code was changed. Source
> citations use `controller/internal/...:line` (a different repo, so links are not
> clickable). Classifications reflect the **target model**: the in-guest controller is
> **Docker-only and holds no Proxmox credentials**; everything host/disk/Proxmox moves to
> a new **host agent** (out of scope here); the controller reaches the agent through a
> constrained **local API**.

## Classification scheme
**KEEP** (host-agnostic, ~unchanged) · **PORT** (survives, needs rework) ·
**DELETE (→agent)** (responsibility moves to the host agent) ·
**DELETE (obsolete)** (no longer needed) · **MODIFY** (stays, materially changes) ·
**NEW** (no v0.33 equivalent).
Risk tags: **clean** · **needs-rework** · **hazard** (entangles a delete-target with a keep/port target).

---

## 0. Executive summary

- The **app domain is largely intact and portable**: stack lifecycle (`stacks/`), catalog
  git-sync (`sync/`), app-to-app integrations (`integrations/`), `.fab` export/import
  (`appexport/`), the scheduler, crypto, asset sync, the hub report/notify *channels*, and
  most of the web UI **KEEP/PORT cleanly**.
- The **disk/storage/host half deletes wholesale to the agent**: all of `storage/`,
  `monitor/watchdog.go`, the restic/cross-drive/disk-layout/drive-mount parts of `backup/`,
  `report/infra_backup*`+`infra_pull`, and the host-physical parts of `system/`.
- The **setup wizard (`setup/`) is obsolete** — the agent provisions the controller.
- **The single biggest hazard is `backup/`**: the keep side (DB dumps, Docker-volume
  archive, per-app restore — needed by `appexport/` and the backup UI) and the delete side
  (restic, cross-drive, drive-mount) are **interleaved inside the same files**
  (`backup.go`, `restore.go`, `paths.go`), not cleanly file-separated. Extracting the
  app-data-backup subset into a clean retained package is the critical refactor.
- **Intent-vs-reality corrections** (vs the task's provisional split): `monitor/pinger.go`
  is already **dead** (legacy Healthchecks.io, "deprecated… now handled by Hub" per
  `main.go`) → DELETE(obsolete), not keep. `backup.go`/`restore.go`/`paths.go` do **not**
  split on file boundaries — they split *within* the file. `settings/` is **not** pure app
  domain — it stores disk/disconnect/decommission state. `system/` is genuinely
  mixed-per-function, not per-file.

---

## 0a. App state: desired / in-flight / observed — S-1 CONTRACT (2026-08-02, decision D-b, R-166)

> **This section is a live contract, not migration history** — the rest of this document is the
> v0.33 keep/port/delete inventory. Read this before touching `stacks/`, `backup/` or `bootrecon/`.
> Shipped in controller **v0.189.0**.

An app's state is **three different kinds of information**, and conflating them is what produced
R-157 mechanism B and F-CRIT-1. They are stored differently on purpose.

| Kind | Question it answers | Where it lives | Persisted? |
|---|---|---|---|
| **Desired** | *What did the customer ask for?* | `app.yaml` → `desired_state` | Yes, beside the app's other settings |
| **In-flight** | *Is an operation part-way through, and did it finish?* | its **own** marker file under `<data_dir>` | Yes, written before the operation and cleared after |
| **Observed** | *Is it running, unhealthy, restarting, is its drive gone?* | nowhere | **No — rebuilt by looking** |

**The rule that ties them together: never derive one from another.** The defect this replaced did
exactly that — it derived *desired* from *observed* (zero containers ⇒ "the customer stopped it"),
and zero containers is equally what a power cut mid-compose, an interrupted deploy and an interrupted
backup leave behind. Two real faults were therefore read as deliberate stops and stranded silently.

### Desired — `app.yaml`, `desired_state`

Tri-state: `""` (unknown) · `"running"` · `"stopped"`.

- **ONE OWNER: the customer's own action.** Writers are the `/api/stacks/{name}/{action}` switch,
  `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, and the `.fab` import. **`StartStack` and
  `StopStack` are NOT writers** — a census found 14 callers of which only 2 are the customer; the
  rest are quiesce, the backup volume dump, offbox reconstitution, app export/restore, the storage
  gate, migration and the boot reconciler. Intent recorded in the primitive would make a nightly
  backup indistinguishable from the customer pressing Stop.
- **Written BEFORE the act; a failed write REFUSES the act.**
- **Absent means UNKNOWN — never "running".** Every `app.yaml` predating v0.189.0 lacks the field,
  so consumers must fall back to the pre-v0.189.0 behaviour rather than assume. A running-only
  backfill converges the unambiguous cases; **`stopped` is never inferred, from any signal.**

### In-flight — a marker file, one per owner

Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
note is a stranded app by a different route.

- Written **before** the stop; cleared **only** after a restart that succeeded; a **failed** restart
  keeps the marker so the next startup retries.
- **A `defer` is not the mechanism.** A SIGKILL runs no deferred function — established on live
  hardware by Campaign 8 fault 10, where what brought the stacks back was the marker read at startup.
- Recovery runs at startup and **completes before** the boot reconciler is launched, so an app the
  marker explains is not also reported as an unexplained boot orphan.

### Observed — not persisted, by design

`aggregateState` walks **every container** of a stack and any unhealthy or mixed result wins, so a
partly-dead app cannot read as healthy (F-CRIT-1's shape). This requirement is met here and must not
be re-implemented downstream. Nothing about observed state is written to disk: a controller restart
re-observes it within one refresh, whereas persisting it risks carrying a stale verdict across the
very restart that fixed it (the same argument as `RestartingSince`).

### The binding safety rule (verbatim, from decision D-b)

> *Losing the state store must never cause an app to be deleted, restarted wrongly, or reported
> healthy when it is not — the worst acceptable outcome is re-running a backup that already ran.*

Applied: a lost or corrupt marker means the app is not auto-restarted **by that mechanism**, which is
the pre-v0.189.0 position, not a new hazard. A lost `app.yaml` already means the app is not deployed.
**Nothing here may make an absent file more dangerous than a present one.**

### Boot recovery reads desired, and asks before it acts (v0.190.0)

**Both boot gates now read desired state.** `bootrecon.isBootOrphan` (the R-52 sweep) and
`shouldRecreateOnBoot` (the drive-backed recreate gate) answer the same question — *did the customer
want this running?* — with the same three-way table, absent falling back to the pre-v0.190.0
container count in both. Their agreement is pinned from both sides against one fixture table, because
an import cycle prevents testing them together. R-170 closed the last gate that still guessed.

**The sweep observes a SETTLED fleet, not a single early sample.** It samples (name, state, container
count) every 5 s, calls the fleet settled after 3 identical samples, and sweeps **once**, at the end.
The window ends on settled or a 50 s budget, and the log says which. Two constraints bound it:

- `bootReconcileSettle + budget + one DefaultRetryDelay` must stay inside `deadAppBootGrace`, or a
  successful recovery stops being silent. This is arithmetic, pinned by a test.
- **each sample must REFRESH first.** `GetStacks()` is the Manager's in-memory map, refreshed by the
  scheduler on its own 10 s cadence; sampling it faster without refreshing lets "settled" mean "the
  cache did not update". Found by live validation, not review.

A recovery completing after the grace emits a `LATE RECOVERY` warning naming the apps. The grace is
**never** widened to make a late recovery look silent.

**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
answer it, and the last two only became reachable once the window widened past T+5 s:

| Holder | Why starting would be wrong |
|---|---|
| the drive-absent gate | compose creates bind sources wherever the mountpoint points — the guest rootfs |
| a quiesce | a running app inside a snapshot meant to be clean-shutdown-consistent |
| an in-flight app-data operation | restarting an app under its own tar |

The rule is **not new** — the API's `startGatedByMissingDrive` already refused a customer's start on
an absent drive. The sweep bypassed it by calling `Manager.StartStack` directly, which is what R-171
closed. Held apps are reported separately from "still down": they are not a fault the sweep failed to
fix, and reporting them as one is a false alarm.

**Read this before adding a fourth caller of `Manager.StartStack`.** That method has no gate of its
own; every caller that is not the customer must decide for itself whether the app may run.

---

## 1. v0.33 module inventory (package → purpose, key deps)

| Package | Purpose | Key internal deps |
|---|---|---|
| `cmd/controller/main.go` | Entry point; wires all subsystems; 6 adapters break import cycles; branches into setup mode | imports **every** package |
| `api/` | REST API (`router.go`) + geo endpoints (`geo.go`) | stacks, backup, metrics, notify, selfupdate, sync, system, assets, integrations, cloudflare, config, settings |
| `appexport/` | `.fab` app export/import (config+DB+volumes, AES-256-CTR+scrypt) | **backup** (DB dump), (provider iface → stacks) |
| `assets/` | Download/cache app assets from Hub API | — (HTTP only) |
| `backup/` | DB dumps, Docker-volume archive, **restic**, **cross-drive rsync**, per-app restore, **drive mount**, disk-layout, infra-backup metadata | config, monitor, settings, system, util |
| `cloudflare/` | Geo-restriction via Cloudflare WAF (zone/waf/geosync/countries) — **enforcement → hub** (S4) | settings |
| `config/` | `controller.yaml` schema + load | — |
| `crypto/` | AES-256-GCM for app.yaml secrets | — |
| `integrations/` | App-to-app (OnlyOffice→FileBrowser/Nextcloud) via docker exec / config patch | stacks, crypto, settings |
| `metrics/` | SQLite time-series: system + container metrics, log scan | system |
| `monitor/` | App health (`healthcheck`,`pinger`) + **storage/USB watchdog** | config, notify, settings, system |
| `notify/` | Hub event push (direct, own API key) | settings |
| `recovery/` | Generate `recovery-info.txt` (DR guide) | — |
| `report/` | Build+push hub report; **infra-backup payload**; **recovery pull** | backup, config, metrics, monitor, scheduler, settings, stacks, system |
| `scheduler/` | Cron/interval jobs, Budapest TZ | — |
| `selftest/` | Startup checks (docker/dirs/catalog/hub/**restic repos**/mountpoint) | backup, config, settings, system |
| `selfupdate/` | Self-update: pull image, edit compose, `up -d` | config |
| `settings/` | `settings.json` persistent state: **storage paths/disconnect/decommission**, cross-drive cfg, notif prefs, geo, integration state, DB-validation cache | — |
| `setup/` | **First-run wizard** (scan drives, hub-restore, manual config) | backup, config, report, settings, web |
| `stacks/` | Docker Compose lifecycle, deploy + memory validation, metadata (`.felhom.yml`), HDD-data delete | config, crypto, system |
| `storage/` | **Physical disk** scan/format/attach/mount/migrate/fstab/safety | backup, settings, util |
| `sync/` | Catalog git-sync (pull templates) | config |
| `system/` | Resource info: mem/cpu/load (guest) + **temp/disk-model/USB/mount topology (host)** | — |
| `util/` | String helper | — |
| `web/` | Hungarian dashboard: pages, auth, deploy, backup UI, **storage/disk UI**, DR restore UI, export UI, debug | appexport, backup, config, crypto, integrations, monitor, notify, scheduler, selfupdate, settings, stacks, storage, system |

---

## 2. Classification table (per package/file)

### `cmd/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `cmd/controller/main.go` | **MODIFY** | Wiring stays, but drop the setup-mode branch, the storage/watchdog/drive-migrator/restic/cross-drive/infra-backup wiring, and add the **agent local-API client**. 6 adapters shrink. | hazard |

### `api/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `api/router.go` | **PORT/MODIFY** | Keep stacks/deploy/integrations/metrics/sync/assets/selfupdate routes; **remove `/api/storage/*` (disk)**; backup routes become **agent-coordinated guest-backup** requests; `config/apply` (hub-pushes-yaml) changes since the **agent** now injects config at provision. | needs-rework |
| `api/geo.go` | **PORT/MODIFY** | Keep the customer-facing geo **preference** endpoints (set/get global + per-app); **drop the Cloudflare-sync trigger** — enforcement → hub (S4). The controller reports geo desired-state up instead of calling the CF API. | needs-rework |

### `appexport/` — KEEP/PORT (Docker-volume + DB level, no disk ops)
| File | Class | Reason | Risk |
|---|---|---|---|
| `crypto.go` | **KEEP** | Self-contained AES-256-CTR+HMAC+scrypt for `.fab`. | clean |
| `manifest.go`, `provider.go` | **KEEP** | Bundle metadata; provider interface (impl in main). | clean |
| `export.go` | **PORT** | Docker-volume `tar`, DB dump via `backup.DumpOne`, config copy. Depends on the **retained** app-data-backup subset of `backup/`; HDD-mount enumeration reworked to **per-volume placement**. | needs-rework |
| `restore.go` | **PORT** | `docker volume create`/`tar xf`, DB import, compose up. Same per-volume rework. | needs-rework |
| `estimate.go` | **PORT** | `du`/`df` on mounts → per-volume sizing. | clean |

### `assets/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `syncer.go` | **KEEP** | Hub API download + checksum cache; already a direct hub channel. | clean |

### `backup/` — THE SPLIT (delete side interleaved with keep side; see §3)
| File | Class | Reason | Risk |
|---|---|---|---|
| `dbdump.go` | **KEEP** | Pure `docker exec pg_dump`/`mariadb-dump` — app/DB data layer; the retained per-app backup. | clean |
| `appdata.go` | **PORT** | App-data discovery (stacks/volumes/DB containers, `du`). "HDD mount" concept → per-volume. | needs-rework |
| `backup.go` (1478 L) | **MODIFY (split)** | Mixes **keep** (`RunDBDumps`, `DumpAppVolumes(Safe)`, app restore) with **delete→agent** (`RunBackup`/`backupDrive`/restic snapshot/prune/check on per-drive repos). Must be torn in two. | hazard |
| `restore.go` (442 L) | **MODIFY (split)** | `RestoreApp` restic path → agent; Docker-volume + Tier-2 rsync restore (app layer) → keep. | hazard |
| `restore_app_linux.go`/`_other.go` | **PORT** | Per-app restore: compose pull/up, rsync app data, DB-dump restore. App layer; depends on backup location that changes. | needs-rework |
| `paths.go` | **MODIFY (split)** | `AppDBDumpPath`/`AppVolumeDumpPath` keep; `Primary/SecondaryResticRepoPath`, `InfraBackupDir` → agent. | needs-rework |
| `restic.go` | **DELETE (→agent)** | restic repos on drives = infra backup tier; agent does vzdump/PBS. | hazard |
| `crossdrive.go` | **DELETE (→agent)** | Tier-2 cross-drive rsync to secondary storage = storage-tier (agent + storage manifest). | hazard |
| `restore_drives_linux.go`/`_other.go` | **DELETE (→agent)** | `lsblk`/`blkid`/`mount`/fstab — pure host disk. | hazard |
| `disk_layout.go` | **DELETE (→agent)** | Disk topology for DR → agent. | clean |
| `local_infra.go` | **DELETE (→agent)** | Per-drive infra-backup metadata → agent. | clean |
| `restore_scan.go` | **DELETE (→agent)** | Scans drives to build a DR restore plan = agent-tier DR. | needs-rework |

### `cloudflare/` — DELETE (→hub): CF-API enforcement moves to the hub (S4)
| File | Class | Reason | Risk |
|---|---|---|---|
| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework |

### `config/`, `crypto/`, `util/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `config/config.go` | **MODIFY** | Drop `BackupConfig` (restic/retention), storage-drive keys, and `InfrastructureConfig.cf_api_token` (→hub, S4); keep customer/paths/web/git/stacks/monitoring/hub/assets/system; **add agent local-API endpoint+token**. | needs-rework |
| `crypto/crypto.go` | **KEEP** | App.yaml secret encryption. | clean |
| `util/strings.go` | **KEEP** | Trivial helper. | clean |

### `integrations/` — all KEEP (pure app-domain)
| File | Class | Reason | Risk |
|---|---|---|---|
| `integrations.go`,`lifecycle.go`,`manager.go`,`onlyoffice_filebrowser.go`,`onlyoffice_nextcloud.go` | **KEEP** | App-to-app via `docker exec` / compose-config patch; no host ops. | clean |

### `metrics/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `store.go`,`logscanner.go`,`telemetry.go`,`types.go` | **KEEP** | SQLite store, `docker logs` scan, container telemetry — app-domain. | clean |
| `collector.go` | **PORT** | Container metrics (`docker stats`) keep; host metrics via `system.GetInfo` (temp, physical disk) become **agent-provided or dropped**. | needs-rework |
| `sysinfo.go`/`sysinfo_other.go` | **MODIFY** | Reads `/host/etc`, `/proc/cpuinfo`, uptime — host static info; in-guest some is meaningful, hardware identity via agent. | needs-rework |

### `monitor/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `healthcheck.go` | **PORT (split)** | Keep guest health (mem/cpu/docker/protected-containers); host health (temp, **physical disk**, storage-path mount status) becomes **agent-fed**. | needs-rework |
| `pinger.go` | **DELETE (obsolete)** | Legacy Healthchecks.io; `main.go` itself marks it "deprecated… now handled by Hub". *(Corrects the task's KEEP/PORT guess.)* | clean |
| `watchdog.go` (902 L) | **DELETE (→agent)** | Storage/USB disconnect monitoring: `umount -l`, `mount -T /host-fstab`, UUID probing, restic-lock cleanup — pure host storage. | hazard |

### `notify/`, `recovery/`, `scheduler/`, `selftest/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `notify/notifier.go` | **KEEP/MODIFY** | Direct hub event channel (own API key) — keep; prune infra event types that move to the agent (`storage_disconnected`, `crossdrive_*`, `disaster_recovery_*`). | clean |
| `recovery/info.go` | **DELETE (obsolete)** | Generates a DR text guide (OS install, docker-setup.sh, hub restore UI); DR is now agent+hub provisioning. | clean |
| `scheduler/scheduler.go` | **KEEP** | Generic cron/interval, Budapest TZ. | clean |
| `selftest/selftest.go` | **PORT** | Keep docker/dirs/catalog/hub checks; drop restic-repo + system-data **mountpoint** checks (→agent). | needs-rework |

### `report/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `pusher.go` | **KEEP** | Direct hub push (`/api/v1/report`, Bearer). | clean |
| `telemetry.go` | **KEEP** | Per-app telemetry section. | clean |
| `builder.go` (326 L) | **MODIFY** | Keep containers/telemetry/stacks/geo/app-health; drop/relocate host system info, physical storage, **restic backup status incl. restic password**. | hazard |
| `types.go` | **MODIFY** | Schema: drop infra fields (`restic password`, physical storage), keep app-domain. | needs-rework |
| `infra_backup.go`/`_linux.go`/`_other.go` | **DELETE (→agent)** | Builds infra-backup payload (disk layout, restic/enc passwords) for hub. | hazard |
| `infra_pull.go` | **DELETE (→agent)** | Pulls recovery config + infra backup from hub (setup-wizard DR). | needs-rework |

### `selfupdate/` — controller is agent-managed (doc 03 §11)
| File | Class | Reason | Risk |
|---|---|---|---|
| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean |
| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean |
| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean |

### `settings/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `settings/settings.go` (1101 L) | **MODIFY (split)** | Keep notif prefs, integration state, geo, DB-validation cache, cross-drive *intent*. The **storage-path registry** (`StoragePath` with `Disconnected`/`DisconnectedAt`/`StoppedStacks`/decommission) is disk-management state → reshape to **per-volume placement** fed by the agent's storage manifest; disconnect/decommission/migrate state leaves. (UUID is *not* a persisted field — runtime-derived from fstab.) | hazard |

### `setup/` — all DELETE (obsolete); the agent provisions the controller
| File | Class | Reason | Risk |
|---|---|---|---|
| `handlers.go`,`setup.go`,`csrf.go`,`network.go` | **DELETE (obsolete)** | First-run wizard (hub-restore, manual config, LAN-IP detection). | needs-rework |
| `scanner.go` | **DELETE (→agent)** | Drive scan (`lsblk`+temp mounts) for backup discovery — host op; its capability informs the agent. | clean |

### `stacks/` — core app domain (KEEP/PORT)
| File | Class | Reason | Risk |
|---|---|---|---|
| `manager.go` (1074 L) | **KEEP/PORT** | Docker Compose orchestration, scan/state/start/stop/logs — the heart. Minor port. | clean |
| `deploy.go` | **PORT** | Memory validation (`system.GetMemoryMB` — **guest** mem, fine in LXC), secret gen, encrypted app.yaml. **Add snapshot-before-deploy → agent** hook. | needs-rework |
| `healthprobe.go` | **KEEP** | TCP/HTTP app probes. | clean |
| `metadata.go` | **PORT** | `.felhom.yml` parse. **Add per-volume hot/bulk classification** (doc 01 §8). | needs-rework |
| `delete.go` | **PORT** | Stack delete + HDD-data `os.RemoveAll` on bind mounts → per-volume cleanup. | needs-rework |

### `storage/` — entire package DELETE (→agent)
| File | Class | Reason | Risk |
|---|---|---|---|
| `scan*`,`format*`,`attach*`,`migrate*`,`migrate_drive*`,`safety*` | **DELETE (→agent)** | Physical disk: `lsblk`/`sfdisk`/`wipefs`/`mkfs.ext4`/`partprobe`/`mount`/`umount`/fstab/`blkid`/drive-rsync. The agent owns all of this (doc 01 §3, §8). | hazard |

### `sync/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `sync/sync.go` | **KEEP** | Catalog git-sync (clone/fetch/reset, copy compose+`.felhom.yml`, never overwrite app.yaml). **It copies into EVERY stack folder, deployed or not — see "the app-definition seam" below.** | clean |

### `system/` — split per-function (not per-file)
| File | Class | Reason | Risk |
|---|---|---|---|
| `cpu_linux.go`/`cpu_other.go` | **KEEP** | `/proc/stat` works inside an LXC. | clean |
| `info.go`/`info_other.go` | **KEEP** | Structs/stubs. | clean |
| `info_linux.go` | **MODIFY (split)** | Keep mem (`/proc/meminfo`)/load/statfs (guest); **temp via `/host/sys`, hwmon → agent**. | needs-rework |
| `mounts_linux.go`/`mounts_other.go` | **DELETE (→agent)** mostly | Mount-point detection, USB, disk model, fstab, probe — host/disk. Guest-meaningful `statfs` disk-usage is the only keep-candidate → fold into the kept `info`. | hazard |

### `web/` — split by UI surface
| File | Class | Reason | Risk |
|---|---|---|---|
| `auth.go`,`csrf.go`,`logbuffer.go`,`embed.go`,`templates.go` | **KEEP** | Session/CSRF, log ring buffer, embeds/logo. | clean |
| `funcmap.go` | **KEEP/PORT** | Template helpers; a few backup/state labels track the backup rework. | clean |
| `server.go` (559 L) | **MODIFY** | Routing/wiring; remove storage/DR-restore/watchdog wiring; keep app/deploy/backup/settings/export/debug. | needs-rework |
| `handlers.go` (1883 L) | **PORT/MODIFY** | Core pages keep; the embedded **storage-path management** (add/remove/label/schedulable, storage bars, FileBrowser mount sync) → per-volume / agent-fed. | hazard |
| `handler_export.go` | **KEEP/PORT** | `.fab` UI. | clean |
| `handler_debug.go` (823 L) | **PORT** | Drop storage-simulate/infra-push/DR debug; keep the rest. | needs-rework |
| `alerts.go` | **PORT/MODIFY** | Storage-disconnect alert now sourced from **agent** status; backup/update alerts keep. | needs-rework |
| `handler_restore.go` | **DELETE (→agent) / MODIFY** | DR restore-mode UI; DR is agent-tier — replace with an agent-status view or remove. | needs-rework |
| `storage_handlers.go` (1600 L) | **DELETE (→agent)** | Format/attach/mount/disconnect/migrate-drive/decommission disk UI. Any survivor is a **thin client calling the agent API** (e.g. per-volume placement requests). | hazard |
| `templates/` (HTML, non-Go) | **PORT** | Remove disk-wizard + DR pages; keep app/deploy/backup/settings pages. | needs-rework |

### `scripts/`
| File | Class | Reason | Risk |
|---|---|---|---|
| `scripts/hashpass.go` | **KEEP** | Standalone bcrypt helper. | clean |

---

## 3. Coupling hazards (delete-targets depended on by keep/port)

1. **`backup/` is half-deleted but split *inside files*, not across them.** `backup.go`
   contains both `RunDBDumps`/`DumpAppVolumesSafe`/app-restore (keep) and
   `RunBackup`/`backupDrive` + restic (delete→agent); `restore.go` and `paths.go` are
   likewise mixed. **Keep/port consumers reach into this same package:**
   - `appexport/export.go:295` → `backup.DiscoverDatabases`/`DumpOne` (DB dump is app-layer — must survive)
   - `report/builder.go:buildBackupReport` → backup status (MODIFY)
   - `web/handlers.go` (backups page, `buildAppBackupRows`), `web/funcmap.go`, `web/alerts.go`, `web/handler_restore.go`, `web/handler_debug.go`
   - `selftest/selftest.go:217` → `checkResticRepos` (restic path — delete)
   - `main.go` scheduler chain `RunFullBackup` (DB→volume→restic→infra-push) interleaves both sides.
   **Action:** extract the app-data-backup subset (DB dump, volume archive, per-app
   restore) into a clean retained package *before* deleting the restic/cross-drive code,
   or every keep consumer breaks.

2. **`backup/crossdrive.go` (delete→agent) is wired as `crossDriveRunner` into**
   `main.go`, `api/router.go`, `web/server.go`, and surfaced by `report/builder.go` and the
   backups page. Removing it requires reworking the backup UI/report to the agent's
   guest-backup status.

3. **`storage/` (delete→agent) depended on by keep/port UI:** `web/storage_handlers.go`
   (delete) and `web/server.go`/`web/handlers.go` (port) — the latter renders storage
   labels/bars and runs **FileBrowser mount sync** off the storage-path registry.
   `storage/migrate*.go` also imports `backup` (also being split). Untangle the per-volume
   placement UI from the disk-management UI.

4. **`monitor/watchdog.go` (delete→agent) depended on by** `web/alerts.go` (port),
   `web/server.go`, `web/handler_debug.go`, `main.go`. The disconnect **alert** must instead
   consume agent-reported storage status.

5. **`system/` mixed-per-function, consumed by both sides.** Keep consumers —
   `stacks/deploy.go` (`GetMemoryMB`, guest), `metrics/collector.go` (container) — must not
   drag in the host-disk/temp/USB code that goes to the agent (`mounts_linux.go`,
   `info_linux.go` temp). Also consumed by `report/builder.go` (MODIFY), `monitor/healthcheck.go`
   (PORT), `selftest`, `crossdrive` (delete). **Split `system/` cleanly into guest-info vs
   host-info first.**

6. **`settings/StoragePath` carries disk state into an app-domain store.** Disk fields
   (`Disconnected`,`DisconnectedAt`,`StoppedStacks`, decommission — UUID is *not* persisted, it's runtime-derived from fstab via `system.ParseFstabUUID`/`watchdog.go`) are written by
   `watchdog.go`/`storage_handlers.go`/`crossdrive.go` (all delete) but the same struct is
   read by `stacks`/`web` for labels and **placement** (keep). Reshape `StoragePath` to a
   placement record fed by the agent manifest.

7. **`report/builder.go` imports almost everything** (backup, monitor, scheduler, stacks,
   system, metrics, settings, config). Its MODIFY must land *after* the backup and system
   splits, or it pulls deleted code along.

8. **`backup/paths.go` shared both ways** — `appexport` + `selftest` + the kept DB-dump
   flow use the app-dump path helpers; the same file holds the restic/secondary helpers
   that leave.

9. **DR/provisioning chain is cross-cut:** `setup/` (obsolete) → `report/infra_pull` +
   `recovery/info` + `backup.MountDrivesFromLayout` + `backup.ReadLocalInfraBackup`. All
   obsolete/→agent, but `main.go`'s setup branch and `web/handler_restore.go` reference
   them; remove together.

---

## 4. Moves to the host agent (consolidated — feeds the future agent design)

> Reporting only; **not** designing the agent here.

- **All physical-disk management** — `storage/` in full: scan/classify, format
  (`wipefs`/`sfdisk`/`mkfs.ext4`/`partprobe`), attach (raw mount + bind + fstab), per-app
  and full-drive migration (rsync), safety checks (system-disk detection).
- **Storage/USB watchdog** — `monitor/watchdog.go`: disconnect/reconnect detection,
  `umount -l`, `mount -T /host-fstab`, UUID-by-id probing, safe-disconnect, restic-lock
  cleanup.
- **Infra/disk backup tier** — `backup/restic.go`, `crossdrive.go`,
  `restore_drives_*`, `disk_layout.go`, `local_infra.go`, `restore_scan.go`, plus the
  restic-snapshot half of `backup.go`, the restic-restore half of `restore.go`, and the
  restic/secondary path helpers in `paths.go`. (Maps to the agent's `vzdump`→tiers→PBS in
  doc 01 §8.)
- **Infra-backup payload + recovery pull** — `report/infra_backup*`, `report/infra_pull`.
- **Host-physical telemetry** — `system/mounts_linux.go` (mount topology, USB, disk
  model), the temp/hwmon parts of `system/info_linux.go`, and the host-hardware parts of
  `metrics/sysinfo.go`.
- **Drive scanning for provisioning/DR** — `setup/scanner.go`.
- **Self-restore-test execution** — the agent performs the restore-to-scratch-guest; the
  controller only orchestrates/validates (see §5).

---

## 5. New components to build (no v0.33 equivalent)

1. **Agent local-API client** — the controller's only path to guest-level Proxmox
   operations (doc 01 §3, §5): `snapshot-before-deploy` + rollback, "grow my RAM", request
   guest backup/restore, read the storage manifest / mount placement, query per-target
   storage status. Replaces the deleted direct host/disk code with constrained RPC. The
   controller holds **no Proxmox creds** — only a local-API token.
2. **Per-volume storage placement** (doc 01 §8) — `.felhom.yml` `hot`/`bulk` volume
   classification (extend `stacks/metadata.go`), enforcement at deploy (extend
   `stacks/deploy.go`), and a placement record in `settings`. Replaces the per-app
   HDD-path + cross-drive model. A `bulk` volume must be realized as a `backup=0` mount point,
   **never** a rootfs Docker named volume (validated recipe: `phase3-findings.md` B2 / doc 03 §7).
3. **Self-restore-test status display** (read-only) — the **agent owns orchestration** (it
   holds the PBS key and creates the scratch guest — operator-tier, doc 03 §8); the controller
   only surfaces `GET /restore-test/status` in its UI. (Round-trip validated: Phase 2,
   [../proxmox-platform.md](../proxmox-platform.md) §4.)
4. **Snapshot-before-deploy/rollback flow** in the deploy path — wraps the existing
   compose deploy with agent snapshot → health check → agent rollback-on-failure
   (doc 01 §9). New behaviour on top of `stacks/deploy.go` + `stacks/healthprobe.go`.
5. **Agent-provisioning bootstrap receiver** — the controller accepts its injected hub API
   key + local-API token from the agent at provision time (doc 01 §6), replacing the
   deleted `setup/` wizard.

---

## 6. Open / blocked items

- **Geo — resolved (S4):** CF-API **enforcement moves to the hub** (it holds the CF token and
  reconciles geo → WAF); the controller keeps the geo **preference UI/data** and reports
  desired-state up. Tunnel placement is settled (host, agent-managed, doc 03 §3/§5). The
  `cloudflare/` package + `api/geo.go`'s CF-sync are DELETE-from-controller → hub.
- **Self-update — resolved (doc 03 §11):** the controller is agent-managed; its self-update
  path is removed.
- **`settings`/`stacks` per-volume reshape** — depends on the storage-manifest contract
  between hub ↔ agent ↔ controller (doc 01 §8), not yet specified.
- **Backup UI/report surface** — depends on the agent's guest-backup status API shape
  (what the controller can see about vzdump/PBS state) — undefined.
- **Notification event taxonomy** — which infra events (`storage_disconnected`,
  `crossdrive_*`, `disaster_recovery_*`) the **agent** emits vs the controller, once those
  responsibilities move.

---

## Changelog — design-review + Phase-3 fold-in (2026-06-08)

- **M1:** removed `UUID` from the `settings.StoragePath` field lists (§ settings, hazard #6) —
  it is runtime-derived from fstab, not persisted.
- **S4 (geo):** `cloudflare/` reclassified **PORT(blocked) → DELETE(→hub)** (CF-API enforcement
  moves to the hub); `api/geo.go` → **PORT/MODIFY** (keep geo *preference* endpoints, drop the
  CF-sync trigger); `config/config.go` also drops `cf_api_token`. §6 + §1 updated.
- **S5:** cloudflare/geo no longer "blocked on tunnel placement" (resolved).
- **S6:** §5(3) self-restore-test → **status-display only**; the agent owns orchestration.
- **Self-update resolved (03 §11):** `updater.go` → **DELETE(→agent)**, `state.go` →
  DELETE(obsolete), `version.go` KEEP; §6 + §5(2) updated (bulk = `backup=0` mountpoint recipe).


---

## The app-definition seam — what happens to a deployed app when its compose file is rewritten

> **STATUS: MEASURED BEHAVIOUR, NOT A RECORDED DESIGN.** This section describes what the system was
> observed to do on 2026-09-01 (`audits/SPIKE-app-update-2026-09-01.md`), with controls. It is
> deliberately **not** marked `[DESIGN]`, because the operator has not ruled on whether the behaviour
> was intended. **Do not read this as an endorsement, and do not spec against it as though it were
> settled.** The open questions are R-438 and R-441.

Until this was measured, no architecture document said what happens here, and the gap itself is
R-438. The three facts below are the ones a reader needs before touching any of it.

### 1. The catalog syncer rewrites the file under a running app, on a 15-minute cycle

`Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies
`docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether
the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351`), plus
one immediate sync at controller start (`sync.go:98`).

**It restarts nothing.** The post-sync hook is `stackMgr.InjectMissingFields(updated)` and nothing
else. So from the moment it runs, a deployed app's *definition* and its *running containers* disagree,
and they stay that way until something else acts.

### 2. Every lifecycle path resolves that disagreement, silently, by upgrading

`StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in
`docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it
is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control
(unchanged file) that did not even recreate the container.

`RestartStack` carries an explicit in-source comment saying this is deliberate — *"so that … any
template changes (new images, healthchecks) are picked up"*. **That is a fact about the source, not an
operator ruling**, and it speaks only for the customer-pressed restart.

**Thirteen non-API call sites across nine files** reach `up -d` without anyone pressing anything. The
full table is §8 of the spike doc; the three that matter most are:

- `bootrecon.Reconciler.Run` → `StartStack` — `bootrecon/bootrecon.go:269`
- `backup.AppStopGuard.Recover` → `StartStack` — `backup/appstop_marker.go:283`
- the drive-return gate, `web.Server.restartStacks` → `StartStack` — `web/intermediary.go:222`

**A plain power cut does NOT trigger this.** Docker's `restart: unless-stopped` restores the existing
containers on the old image, the reconciler finds no orphan, and it logs so
(`no boot-orphaned apps (nothing to start)`). The unattended upgrade needs the narrower precondition
*"and the app did not come back"* — which was measured, and does upgrade.

### 3. Nothing takes a copy first, and the tag cannot be put back afterwards

`UpdateStack` runs `compose pull` then `compose up -d --remove-orphans` and does nothing else — no
dump, no copy, no hold. The R-361 safety dump (`backup/offbox_reconstitute.go:207`) is **database-only**
and is not on this path at all; an app with no database gets nothing from it even on the paths where it
does run.

**And restoring the old tag is not a rollback.** Once an app has migrated its data, the old image
refuses to start — measured on Nextcloud: *"the version of the data (32.0.9.2) is higher than the
docker image version (31.0.14.1) and downgrading is not supported"*. The data itself survives; only the
downgrade is blocked. So the only route back is a **data** restore from a copy taken before the
update.

**And that route fights this seam** (R-441): `stackAdapter.RecreateStackDefinitionFromUnit`
(`cmd/controller/main.go:2570`) writes the recovery unit's captured compose — with the OLD pin — into
the live stack dir, and `copyIfChanged` overwrites it again on the next tick. The overwrite is
measured; that the restore writes to that path is read, not measured.

### What a change here must not break

- The syncer's overwrite is also the **repair** path: a locally-corrupted compose is replaced by the
  catalog's version within 15 minutes (observed). Any "don't touch deployed apps" rule loses that.
- `up -d`-on-restart is what injects `app.yaml` env into a running stack. Reverting to
  `docker compose restart` would silently stop doing that.