## NINE DEPLOYS FAILED — and the product said so, correctly, within two minutes
## 2026-09-16, box tester-1-022354 / guest 9201, controller 0.245.0

### What actually happened (root cause, measured)
Ten deploys were admitted at 20:30:19, each passing the controller's memory check, which counts
COMMITTED memory rather than memory in use:
    total=4096MB reserved=384MB usable=3712MB — committed_used climbed 2514 -> 3626MB
    the 11th and 12th (mealie, adventurelog) were REFUSED with the numbers in the message
Then the image pulls began, and at **20:34 the hub recorded**:
    `storage_fill_critical` (critical) — „Host tester-1-022354: storage \"local-lvm\" CRITICALLY full
     at 100% (threshold 95%) — backups/writes to it will fail; free space immediately"
Between 20:32 and 20:33, **nine `app_deploy_failed` (warning) events** were pushed, one per app, each
quoting the failing pull: Gokapi, Paperless-ngx, Vaultwarden, Immich, BookStack, Nextcloud, Homebox,
Jellyfin, Uptime Kuma. Only **PrivateBin** completed (86 s, `app_deployed` at 20:31:45).

The disk is thin-provisioned: the VM's 32 GB system disk yields an ~11.8 GB LVM thin pool, over which
the guest's 32 GB rootfs and 70 GB data volume are over-subscribed. Ten simultaneous image pulls
filled the pool.

### What this says about the PRODUCT — it behaved, and two of tonight's own concerns are answered
  * **R-536's `app_deploy_failed` works.** That event shipped this morning precisely so an
    interrupted install is not silence. Nine interrupted installs produced nine warnings, each
    naming the app and the failing image, within ~2 minutes of the failure. Before R-536 this was
    silence, and the customer would have been left with nine cards that never resolved.
  * **The fill alarm fired at CRITICAL severity** with an actionable sentence, at 95 % threshold.
  * **The memory guard refused rather than over-committing**, and quoted both numbers it compared.
  * The box's own per-stack record is honest: `deployed: false` for all nine, `true` only for
    privatebin. Nothing claims to be installed that is not.

### What this says about MY HARNESS — two errors, both mine
  1. **Twelve deploys fired in two seconds is not household behaviour.** A household installs an app,
     waits for it, then installs another. Firing them in parallel is what drove committed memory to
     the ceiling and ten image pulls onto one thin pool at once.
  2. **The drill VM's system disk (32 GB) is too small for a twelve-app household.** That size was
     copied from a previous drill's VM without checking what that drill actually installed.

Neither is a product defect and neither is recorded as one. The recovery, and the re-seed done one
app at a time, are recorded next.
