# REPORT — felhom-agent v0.29.x (OS / Docker-data storage split + lanresolver fix)

> **Follow-up v0.29.1 (lanresolver):** the v0.29.0 re-provision moved 9201's DHCP IP (.151 → .141), but
> LAN clients kept resolving the old IP. Root cause: `lanresolver` updated the dnsmasq drop-in
> (`address=/<domain>/<ip>`) then ran `systemctl reload dnsmasq` — **SIGHUP does not re-read config
> drop-ins** (only `/etc/hosts` + cache), so the changed `address=` never took effect. Fixed:
> `reload()` → `restartDnsmasq()` (`systemctl restart`). Deployed live; host dnsmasq restarted + Pi-hole
> cache flushed; `*.demo-felhom.eu` now resolves to .141 and LAN HTTPS returns 200. Future IP moves
> self-heal on the loop's next tick. (DHCP for the guest is fine — no static IP needed.)

---

# REPORT — felhom-agent v0.29.0 (OS / Docker-data storage split: golden + provision)

Phase 1 of the storage-split slice (Phase 2 = felhom-controller v0.58.0 prevention layer). The
controller guest's OS rootfs and Docker data are carved onto separate `local-lvm` volumes for
**resilience** — an isolated OS rootfs stays bootable + agent-recoverable if the Docker volume fills.
Built, the golden re-baked, and **live-validated by destroying + re-provisioning guest 9201**.

## What shipped (v0.29.0)
- **`configs/build-golden.sh`** — split baked in: `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` (default
  **32**, was hardcoded 8) **plus** `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
  (default 16). `daemon.json` bakes the **classic overlay2 driver** (`features.containerd-snapshotter:
  false`) + log rotation (`max-size 10m`, `max-file 3`). Guards: aborts unless `/var/lib/docker` is a
  separate mount, the driver is overlay2, and vzdump **includes** mp0 (the B3 trap).
- **`internal/reconcile/bringup.go`** — `GuestMount.Backup` emits `,backup=1` (closes the spike-B3/B5
  silent-DB-loss trap at the mount builder). `BringUpSpec.{DataVolGrowGB,DataVolMount}` grow the
  golden-carried Docker-data volume (default `mp0`) online to the per-customer target rather than
  attaching a fresh empty volume that would shadow the baked images. Plus `RootfsGrowGB`.
- **CLI:** `--selftest=bring-up|provision` gain `-rootfs-grow` / `-datavol-grow` / `-datavol-mount`.
- **`RUNBOOK-provisioning-storage.md`** — provisioning procedure + fresh-PVE-install thin-pool carving
  (`hdsize`/`maxroot`/`maxvz`) + the per-customer sizing seam (flags now; hub storage manifest later).
- Tests: `buildBringUpConfig` backup=1 emission; bring-up issues rootfs + data-volume resizes.

## The critical fix validation caught (overlay2)
Docker 29's **default containerd-snapshotter keeps the image store at `/var/lib/containerd`** (on the OS
rootfs), so mounting the data volume at `/var/lib/docker` only moved named volumes — **1.2 GB of images
stayed on the rootfs** (validated live), defeating the split and breaking the controller's `statfs("/")`
guard. Fix: the classic **overlay2** driver stores everything (images + overlay + volumes) under
data-root = the data volume. A start-vs-restart trap (docker-ce auto-starts on install) was also fixed
(daemon.json needs a `restart`). After the fix, a provisioned guest showed images on the data volume
(`/var/lib/docker/overlay2`), `/var/lib/containerd` idle, and a lean OS rootfs.

## Live validation
1. **Re-baked the golden** with the split (overlay2): docker OK on overlay2, `/var/lib/docker` a separate
   ext4 mount, vzdump "including mount point mp0 ('/var/lib/docker')", 580 MB archive carrying the images.
2. **Throwaway provision (9301)** from the golden: booted in 39 s; `pct config` mp0=`/var/lib/docker,backup=1,size=256G`,
   rootfs 32G; overlay2; baked images on the data volume; **OS isolation** — filled the data volume to
   100%, the OS rootfs stayed at 4% + writable; log rotation effective.
3. **Destroyed + re-provisioned the live 9201** from the same golden via `--selftest=provision`
   (customer `demo-felhom`, retrieval passphrase reused from its old bootstrap; `-datavol-grow 240` →
   256 GB) + a reboot: bring-up booted in 39 s, the back-half minted a token + attached the bootstrap
   mp9, and after reboot the controller-bootstrap deployed the baked controller, which pulled its config
   from the hub. All four infra containers came up **from baked images with no pull**. Split + overlay2 +
   lean rootfs confirmed on the real guest; the controller's deploy gate refused (507) on the full data
   volume. (Phase-2 details in the controller REPORT.)

## Provisioning facts confirmed (for the spec/runbook)
- The restore carries all mountpoints (golden mp0 travels in); `freeMountSlot` picks the lowest free slot
  for a later USB enroll, so **mp0 = Docker-data, mp9 = bootstrap, USB = mp1+** — no collision.
- Attaching/flag-changing a mountpoint on a running unprivileged guest is **pending until reboot** — the
  provision flow's post-provision reboot activates the bootstrap mp9 (and would activate a USB bind).
- `backup=1` on the Docker-data mp is mandatory twice over: keeps named-volume DBs in PBS, AND captures
  the volume (with baked images) into the golden archive.

## Outstanding (demo restoration, not slice validation)
RomM (the HDD app) + USB re-enroll: its data is safe on the host USB (`/mnt/felhom-usb/felhom-data`,
untouched). Restoring it is the slice-10 enroll flow (assign → guest-attach → reboot → register storage
→ deploy); with the split it binds to mp1+ (mp0 is the Docker-data volume). Documented as the final
restore step.