From 41dbecb264b3af13981d30a1820e28fd8d08fb50 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 2 Aug 2026 23:56:16 +0200 Subject: [PATCH] docs: R-167 + R-158 CLOSED, R-165 SPIKED, R-174..R-177 filed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R-167/R-158 shipped and proven live (controller v0.191.x, hub v0.89.0): two new capability-map rows PROVEN-LIVE with live citations, and 07-backup-architecture.md §7.5's closing claim "nothing warns when an app crosses the line" is now false and rewritten (S-1: an architectural contract changed in the same session). §7.5 also gains the caveat that its size bound is ONE BOX'S, not the fleet's. Part 3 SPIKE (audits/SPIKE-r165-mp1-merge-2026-08-02.md): M1-M5 measured, NO layout touched. Three findings the merge session must not re-derive: "the layout" is not one thing (200G/50G vs 50G/20G vs 16G/8G); mp1 is a BULKHEAD and not only a ceiling, so after the merge an overflow reaches /var/lib/docker; the golden fails closed on the split in four places. D-a's condition (1) is currently SATISFIED — no external box is in the hub's register, and both demo boxes are Tier 0 and reinstallable. Recommendation given, choice NOT made — it ends at the operator's ruling. CONTEXT.md S-11 (D-c's routing, and why R-158's own backup_failed proposal was overruled) and S-12 (the monitoring landed BEFORE the merge). STATUS.md gains the plain-language section and the merge decision, with two older entries trimmed so the page did not grow. New rows R-174 (closed same session), R-175, R-176, R-177; each ID grepped free before minting. --- CONTEXT.md | 34 ++ REPORT.md | 352 ++++++------------ STATUS.md | 59 ++- .../architecture/00-capability-map.md | 2 + .../architecture/07-backup-architecture.md | 22 ++ .../audits/SPIKE-r165-mp1-merge-2026-08-02.md | 248 ++++++++++++ documentation/backlog/OPEN-ITEMS.md | 10 +- documentation/backlog/ROADMAP.md | 4 +- 8 files changed, 478 insertions(+), 253 deletions(-) create mode 100644 documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md diff --git a/CONTEXT.md b/CONTEXT.md index 5a27202..ddeef93 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -17,6 +17,40 @@ ## Standing rulings +**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).** +Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the +CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure +is the OPERATOR's** (they can do none of those things about it). R-158 proposed emitting the existing +`backup_failed` for the capture failure. **That was rejected and D-c wins**, because `backup_failed` +carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents` — so reusing it emails +the customer, in Hungarian, that their backup failed, about something they cannot act on. It is +exactly the mistake R-97a avoided by minting `whole_guest_backup_failed`, and the reasoning is written +into `hub/internal/api/handler.go`'s allowlist. New type: `recovery_unit_capture_failed`, in +`allowedEventTypes` **and** `notify.operatorOnlyEvents` — **the second register is what makes it +operator-only; the first does not**, and v0.78.0 claimed otherwise and shipped the defect. + +**The customer half reused the pair that already existed rather than minting a seventh type.** +`disk_warning`/`disk_critical` were allowlisted, carried Hungarian copy, sat in `DefaultEnabledEvents` +and had a UI checkbox — and **nothing in any repo emitted them**. A complete customer pipeline with no +producer: the **sixth** *built-but-never-wired* instance in this project. `internal/fillwatch` is now +that producer. Their generic `customerMessages` entries were **deleted**, because +`FormatCustomerEmail` PREFERS the entry over the message and a static template would discard the drive +label and the free-space figures — the same reason `offbox_enlarge_blocked` and `disk_health_degraded` +have none. `notify.IsOperatorOnly` was added so ONE test pins both registers; checked separately, an +allowlisted-but-not-operator-only type is invisible. + +**S-12 — the monitoring landed BEFORE the merge, not with it (2026-08-02).** +D-a's condition (2) says R-167 ships in the same step as the `mp1`→`mp0` merge and never after, +because the merge removes a wall that currently fails safely. **This session landed it FIRST**, which +is strictly better and costs nothing: the warnings went in and were proven on real hardware while the +wall is still standing, so the merge session inherits a proven signal instead of an untested one. +**No disk layout was touched.** R-165's measurement is `audits/SPIKE-r165-mp1-merge-2026-08-02.md`, +which STOPS at a question for the operator (which merge shape; what replaces the bulkhead). Its two +load-bearing findings for anyone picking that up: **"the layout" is not one thing** (demo-felhom +`200G/50G`, demo-hp `50G/20G`, golden `16G/8G` — so §7.5's bound is one box's, → R-175), and **`mp1` +is also a BULKHEAD**, not only a ceiling — today an overflow cannot reach `/var/lib/docker`, and after +the merge it can. + **S-8 — CI detects; it does not block, and that is structural (2026-08-02, R-168).** A Gitea Actions runner in `gitea-system` re-runs every repo's gate entry point on every push, independent of who pushed and of what they typed. It **cannot refuse a push**: every felhom repo diff --git a/REPORT.md b/REPORT.md index 16ea34d..b08c83f 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,250 +1,142 @@ -# REPORT — CI: a Gitea Actions runner, and a red run that reaches a person (R-168, 2026-08-02) +# REPORT — hub v0.89.0: the two halves of decision D-c, plus the R-165 merge spike (2026-08-02) -**Overwritten** per the standing rule. The prior contents (session 1's gate enforcement, same day) -have their durable record in `scripts/CHANGELOG.md`, `hub/CHANGELOG.md` and `CONTEXT.md` S-6/S-7. +**Overwritten** per the standing rule. The prior contents (R-168, the CI runner, same day) have their +durable record in `scripts/CHANGELOG.md` and `CONTEXT.md` S-8/S-9/S-10. -**Outcome: complete, and the half that matters is demonstrated.** All six probes answered; none -STOPped. The acceptance test was never a green pipeline — it was a **red run with a message in -hand**, and that is `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. +**Companion report:** `felhom-controller/REPORT.md` holds the controller side (v0.191.0/.1/.2), the +full red-proof table, the Hungarian copy, and the live evidence for all three flows. This file covers +the hub change, the documentation coupling, and **Part 3's spike**. --- -## 1. Confirmed baselines — all matched §1 on arrival +## 1. Baseline drift — recorded, because the task's §1 was wrong -| Repo | `main` on arrival | matched? | version | +The task targeted hub **v0.87.0 → v0.88.0**. On arrival `main` was at `8ef92a3f` with hub **v0.88.0 +already shipped** (R-172, the WAL fix), not `d5774d318941`/v0.87.0. Target corrected to **v0.89.0**. +Highest register ID in use was **R-173**, not R-171. + +## 2. Hub change (v0.89.0) + +**One new event type, not two.** The task called for a new customer-facing type *and* a new +operator-only one. Reconnaissance found `disk_warning`/`disk_critical` already allowlisted here, with +Hungarian `customerMessages`, in the controller's `DefaultEnabledEvents` and behind a UI checkbox — +**and with no producer in any repo.** The operator chose to wire that inert pair rather than mint a +near-duplicate, so only the operator type is new. + +| Change | File | Why | +|---|---|---| +| `+ "recovery_unit_capture_failed"` | `internal/api/handler.go` (`allowedEventTypes`) | without it the controller's POST 400s and the event vanishes | +| `+ "recovery_unit_capture_failed"` | `internal/notify/dispatcher.go` (`operatorOnlyEvents`) | **this** is what makes it operator-only; the allowlist does not, and v0.78.0 claimed otherwise and shipped the defect | +| `- customerMessages["disk_warning"]`, `- ["disk_critical"]` | `internal/notify/templates.go` | `FormatCustomerEmail` PREFERS the entry over the message, so a static template would discard the drive label and the free-space figures the controller now sends. Same reason `offbox_enlarge_blocked` and `disk_health_degraded` have none | +| `+ func IsOperatorOnly` | `internal/notify/dispatcher.go` | lets the `api` package pin BOTH registers in ONE test; checked separately, allowlisted-but-not-operator-only is invisible. Read-only — the register stays unexported so nothing can widen it at runtime | +| `REUSE.md` §5 "new event type" rewritten | `REUSE.md` | it told readers to always add a `customerMessages` entry, which is **wrong** for operator-only types and **harmful** for dynamic-message ones | + +**Tests 574 → 579**, full suite green (`go build ./... && go vet ./... && go test ./...`), all five +`repo_gates.py` gates OK. + +**Red-proof (Scenario G), demonstrated not argued:** removing `recovery_unit_capture_failed` from +`operatorOnlyEvents` fails two tests, one reading *"a customer was emailed the OPERATOR-ONLY +recovery_unit_capture_failed (customer@example.com)"*. The dispatch test runs under the **breaking** +configuration — the customer has the event enabled and an email set — because that is the only +configuration in which the missing entry is visible. + +**Live (guest 9201 → hub):** both event types accepted and stored; `operator | sent`; and the positive +observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from +`notification_log`. The customer half: `customer | disk_warning | sent` and `customer | disk_critical +| sent` with the dynamic Hungarian intact. + +**Deploy:** GitOps only — `manifests/hub.yaml` bumped 0.88.0 → 0.89.0 (`6d359a5`), pushed, then a +deliberate ArgoCD hard-refresh + sync. Never `kubectl set image`. App `felhom` **Synced / Healthy**, +`deploy/hub` rolled out, running `gitea.dooplex.hu/admin/felhom-hub:0.89.0`, startup log clean. + +## 3. Part 3 — the R-165 spike. **M1-M5 each answered; nothing was changed.** + +Full document: `documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md`. No partition was created, +resized, moved or deleted; no golden rebuilt; no guest config edited. `ep0` and Peti's box were not +contacted (D-d, `runbooks/target-selection.md`). + +**M1 — what is actually there. ANSWERED, and it contradicts the architecture doc.** + +| | demo-felhom | demo-hp | golden default | |---|---|---|---| -| `homelab-manifests` | `420e8197769b` | yes | n/a | -| `felhom.eu` | `af2d10388021` | yes | hub `v0.87.0` — **no bump** | -| `felhom-controller` | `eaded79b1894` | yes | `v0.188.0` — **no bump** | -| `felhom-agent` | `2c4efed5de32` | yes | `v0.119.0` — **no bump** | -| `app-catalog-felhom.eu` | `f16f29757e95` | yes | n/a | +| `mp0` `/var/lib/docker` | **200 G** (13 G used) | **50 G** (5.4 G used) | 16 G | +| `mp1` `/mnt/sys_drive` | **50 G** (2.0 G used, 5%) | **20 G** (92 M used, 1%) | 8 G | -Live Gitea re-confirmation, **all matched**: version **1.26.2**; Actions enabled on **all five** -repos; **0 runners**; **0 workflow runs ever**; **0 branch protections**; **no `.gitea/`** in any -repo. ArgoCD `gitea` app: `path: gitea-system`, auto-sync **off**, Synced at `420e819`, Healthy. -The only new artifact is the runner image, at its own `0.1.0`. +§7.5 documents the appliance as `mp0 50G / mp1 20G` — that is demo-hp exactly and **not** demo-felhom. +Any merge plan expressed as a fixed pair is already wrong for one of the two boxes that exist. §7.5's +headline bound (*"≈ 19 GB … ≈ 10 GB"*) is derived from `mp1 = 20 G` and is therefore one box's, not +the fleet's → **R-175**, filed and §7.5 annotated in this session. -## 2. All six probe results +**M2 — what lives on `mp1`. ANSWERED, and it is not only backups.** Four things would move: +Tier-1 units of driveless apps (269 M, ~30 apps on demo-felhom), **Tier-2 mirrors (1.7 G — i.e. the +MAJORITY is Tier 2, not Tier 1)**, the `userdata/import` drop zone which lives on the system drive by +**contract** (R-75), and the system-data userdata namespace. Observed fill is 5% / 1%: the constraint +is a **ceiling** problem, not a current-fill one. -Full method and measurement: `documentation/audits/SPIKE-ci-runner-2026-08-02.md`. +**M3 — which merge shapes exist. ANSWERED for three shapes, with ONE item explicitly unmeasured.** +The golden **fails closed on the split in four places**, not one (`build-golden.sh:126,130` +separate-mount asserts + `:315,319` vzdump-exclusion guards). The archive scope `rootfs+mp0+mp1` stays +complete after a merge (the data moves onto `mp0`). `mountParity` holds for new archives. **Unmeasured +and reported as such:** whether a *pre-merge* archive restore-tests into a *merged* guest — reading +`mountParity` says it should, but that is reasoning from source about an unvalidated mechanism, which +this project has got wrong four times → **R-176**. -| Probe | Ruling | Measurement | -|---|---|---| -| **P1** runner picks up a job | **PASS** | run #1 claimed by `felhom-gates-runner`, `P1-OK`, `🏁 Job succeeded` | -| **P2** stock image runs our checks | **known branch → build an image** | stock `act_runner:0.6.1` (Alpine 3.23): `git 2.52.0` PRESENT, `python3` **ABSENT**. Host mode itself fine → privileged/dind never reached for | -| **P3** source without JS actions | **PASS** | `pushed sha == checked-out sha` (`bbd62319…`), plain `git clone`/`fetch` from the in-cluster Service, no `uses:` | -| **P4** one registration, four repos | **PASS** | tasks 7,8,9,10 across four repos, all `runner_id=2 felhom-gates-runner` | -| **P5** does a failure signal anything | **NOTHING SENT → build the alarm** | 0 mail/SMTP/notification log lines, **0** notification rows, after a real `failure` run | -| **P6** persistent state needed | **PVC is load-bearing** | PVC intact → same registration; `/data/.runner` removed → **second record** appears and the first is orphaned | +**M4 — the bulkhead. ANSWERED, and it is the important one.** `mp1` is not only a ceiling: today an +overflow is refused per app with the last good unit byte-identical **and cannot reach +`/var/lib/docker`**. After the merge it can, and a full Docker data-root is a stopped box, not a slow +one. Four replacements costed — a reserved block percentage, **a refusal threshold in the capture +path**, a project quota, or deeming R-167's warnings sufficient — with the trade-off of each. +**Deliberately not chosen: this is the operator's ruling.** -**P5's honest limit, stated rather than glossed:** I did not independently prove Gitea's SMTP path -can *deliver*, so "no mail" cannot be split with certainty between *no action-failure notification in -1.26.2* and *a broken mailer*. It does not change the design — the alarm rides Resend, a different -path — and Scenario C proves that path end to end. +**M5 — existing boxes. ANSWERED for the measurable population; one part honestly UNMEASURED.** The +hub's `/hosts` register holds four hosts, **two ONLINE**, both demo boxes — and both are **Tier 0, +therefore reinstallable rather than migratable (D-d)**, so migration cost for the measurable population +is **zero**. **D-a's condition (1) — "before any external install" — is currently SATISFIED**, which +makes this the cheapest this decision will ever be. **`peti-felhom` exists as a customer with NO host +in the register**, so its layout is not knowable from the hub and the box was not contacted; whether it +needs converting or reinstalling is the operator's information. The in-place migration procedure has +**never been rehearsed**, so "is the box restorable at every point of it?" is currently unknown → also +**R-176**. -## 3. Files created / modified +**Ranked options and recommendation:** (1) **S1 — one volume with the two paths as directories — plus +B2, a refusal threshold in the capture path**, shipped as a fresh-install shape with the demo boxes +reinstalled; (2) S1 + warnings only; (3) S3, grow `mp1` and keep the split (D-a's rejected baseline, +measured for comparison); (4) S2, two mounts on one pool — **not recommended at all**, it satisfies +every assertion while delivering none of the benefit and converts a clean per-app refusal into a +shared-pool exhaustion neither `df` can see coming. -**`homelab-manifests`** — `gitea-system/act-runner.yaml` (NEW: PVC + Deployment), -`gitea-system/act-runner/Dockerfile` (NEW), `CHANGELOG.md`, `REPORT.md`. +**STOPPED at the operator's question**, per the task. The merge is next session's supervised work. -**`felhom.eu`** — `.gitea/workflows/gates.yml` (NEW), -`documentation/audits/SPIKE-ci-runner-2026-08-02.md` (NEW), `CLAUDE.md`, `CONTEXT.md` (S-8…S-11), -`scripts/CHANGELOG.md`, `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`, -`STATUS.md`, `REPORT.md`. *(`hub/internal/web/configs.go` was broken and restored within the session -for Scenario B; it is byte-identical to its starting state — verified by the gate that objects to it.)* +## 4. Documentation coupling -**`felhom-controller`**, **`felhom-agent`**, **`app-catalog-felhom.eu`** — -`.gitea/workflows/gates.yml` (NEW), `CHANGELOG.md`, `REPORT.md`. - -## 4. Commits pushed to `main` - -| Repo | Hash | What | -|---|---|---| -| `homelab-manifests` | `c51b77c` | the runner manifest + Dockerfile | -| `felhom.eu` | `bbd6231` | the temporary Part 0 probe workflow | -| | `666a34d` | real `gates.yml`, probe workflow **deleted** | -| | `3252d51` | **Scenario B** deliberate breakage (`--no-verify`) | -| | `dd13f63` | alarm step v1 — died on `curl: command not found` | -| | `f7dbc33` | alarm rewritten in python3/urllib | -| | `9530de7` | alarm User-Agent (Cloudflare 1010) — **the run that produced the accepted-id** | -| | `4cc1238` | **breakage reverted**, main green | -| `felhom-controller` | `dcc400e` | `gates.yml` (with sibling clone) | -| `felhom-agent` | `eb99144` | `gates.yml` (with sibling clone) | -| `app-catalog-felhom.eu` | `aa57588` | `gates.yml` (`--fast`, no sibling needed) | - -Plus the documentation commits listed at the end of this session. - -## 5. Scenario results A-G - -| # | Result | Positive observable | -|---|---|---| -| **A** | **PASS** | run #2 (`666a34d`) SUCCESS. **Every member gate's own stdout in the CI log**, not just the summary: `site gates OK — BOM, emoji=0…`, `hostinstall gates: ALL PASS`, `hub confirm gate OK…`, `manifest bearer gate OK…`, `OK [hostexecutor]: 61 cited paths — exact 60, suffix 1…`, then `all felhom.eu gates OK` | -| **B** | **PASS** | run #3 (`3252d51`) **FAILURE**. The failing gate's own text: `FAIL: hub/internal/web/configs.go:34 carries const hostInstallVersion…`, `hostinstall gates: 3 FAILURE(S)`, `CONVICTED: hostinstall`, `🏁 Job failed` | -| **C** | **PASS** | run #6 FAILURE → **`RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`**. See §6 | -| **D** | **PASS** | four repos, four runs, **one runner**: tasks 7/8/9/10 all `runner_id=2 felhom-gates-runner`; conclusions `success` ×4 | -| **E** | **PASS** | live pod spec (§8) **and** the refusal in a job log: `/root/.cache/act/…/3.sh: line 2: docker: command not found` → `E-OK docker refused` | -| **F** | **PASS** | catalogue run: `image-pin gate OK — 53 templates, 0 unpinned images`; `--fast SKIPPED: image-resolvable, volume-persistence…`; resolvability/volume-persistence/canary output lines **0**, image-pull lines **0** | -| **G** | **PASS** | P6: PVC intact → runner list unchanged (`1 felhom-gates-runner`); state removed → **`1` and `2`**, i.e. a second registration. Orphan deleted, `204` | - -**Scenario A note worth keeping:** the controller and agent runs prove more than "it ran" — their -reuse tallies in CI (**126 exact / 6 suffix / 1 cross-repo**; **88 / 1 / 1**) are *identical* to the -local pre-push hook's. **CI and the hook agree.** Per §12 a disagreement would have outranked this -whole task; there is none. - -## 6. Scenario C evidence in full - -The run was genuinely red for a genuinely broken gate, and the message is the observable — not a -tick in a UI: - -``` -CONVICTED: hostinstall -RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab -🏁 Job failed -``` - -The Resend id is the **provider's accepted-id**, returned by `api.resend.com` for a message -`monitoring@felhom.eu → admin@felhom.eu`, subject `[felhom CI] gates FAILED in admin/felhom.eu`, -carrying the repo, the commit, a link to the run, and the line that matters most: *if the local -pre-push hook was green for this commit, CI and the hook disagree, and that outranks whatever the -push was for.* - -**The key never appeared in the log** — Gitea masked it (`RESEND_API_KEY: ***`) and a grep for the -key prefix across the run log returns **0 lines**. - -## 7. Runner deployment - -- **GitOps only.** `act-runner.yaml` committed to `homelab-manifests` `main` (`c51b77c`), pushed, - then a **deliberate ArgoCD sync** of the `gitea` Application. No `kubectl apply`, no - `kubectl set image` at any point. -- **ArgoCD:** `sync=Synced health=Healthy`, revision `c51b77c68f10…`. -- **Pod:** `act-runner-…` `1/1 Running`; PVC `act-runner-data` **Bound**, 5Gi, `longhorn`. -- **Running image:** `gitea.dooplex.hu/admin/felhom-act-runner:0.1.0`, verified by deleting the local - copy and **re-pulling from the registry** (`Python 3.12.13`, `git version 2.52.0`) rather than - trusting the push's own output. -- **Online in Gitea's runner list**, with its label: - ``` - id=2 name=felhom-gates-runner status=online labels=[felhom-gates] - ``` - Owner-scoped (`owner_id=1`, `repo_id=0`) — which is why one registration serves four repos. - -## 8. Security posture — shown, not asserted - -From the **live** pod spec: - -``` -privileged : False -allowPrivilegeEscalation : False -capabilities.drop : ['ALL'] -automountServiceAccountToken : False -hostNetwork/hostPID/hostIPC : None None None -nodeSelector : None -volumes : [('data', ['persistentVolumeClaim'])] -hostPath volumes : NONE -docker socket mounts : NONE -serviceaccount token mounts : NONE -resources : requests 50m/128Mi, limits 500m/512Mi -``` - -And the negative proven rather than inferred — from a job log: - -``` -/root/.cache/act/187d434f3fdbb3b4/act/workflow/3.sh: line 2: docker: command not found -E-OK docker refused (output above is the refusal) -``` - -Resources are **half of Gitea's limits** (Gitea: 1 CPU / 1Gi) and a fifth of its requests, -deliberately: Gitea shares this node and holds every repository. The heaviest entry point was -measured at **11.9 MiB RSS, 0.98 s wall**. - -## 9. Teardown — all three layers - -1. **Workload** — nothing to remove: the runner is the deliverable and stays. No scratch Deployment - or PVC was created beyond it. -2. **Host** — no scratch Longhorn volume was created beyond the runner's own 5Gi PVC, which is - intended and in git. Local Docker images (`gitea/act_runner:0.6.1` base and the built - `felhom-act-runner:0.1.0`) remain in DooPlex's local Docker store; they are ordinary build - artifacts and **no global prune was run** (forbidden on this host). -3. **The Gitea-side registration records — the layer this project has missed four times.** - - The **P6b orphan** (`id=1`) was deleted **immediately when it was created**, not left for the - end: `DELETE /api/v1/admin/actions/runners/1 → 204`. - - **Final runner list: exactly one, the intended live runner** — - `id=2 felhom-gates-runner status=online labels=[felhom-gates]`, and the live pod's own - `/data/.runner` carries `id=2`, so the record and the process agree. - **A trap worth recording, because it briefly looked like a failed teardown:** a raw - `select id,name from action_runner` returns **two** rows. Gitea **soft-deletes** runners — the - deleted row is tombstoned (`deleted` set), not removed, and the admin API correctly reports one. - The query without the tombstone filter was the misleading instrument, not the teardown. Same - class as the `| tail -5` census bug and the `go test -run` filter: **a query that does not model - the deletion semantics is not a check.** - - The **temporary probe workflow** `.gitea/workflows/probe.yml` was deleted from `main` in - `666a34d`, the same commit that added the real one. - - The **admin API access token** minted for this session (used to read run conclusions and to - delete the orphan runner) is **revoked**. The clean route — `DELETE /api/v1/users/admin/tokens/ - {id}` — needs HTTP **Basic** auth with the operator's Gitea password, not the token itself; - tried `GET /api/v1/users/admin/tokens` with the `PASSWORD` entry from `~/.config/credentials` - and got **401**, so that is not the Gitea admin password. Revoked instead by deleting the single - `access_token` row I created, matched on its `cc-ci-session-*` name: **1 row before, 0 after**, - 6 unrelated tokens untouched. The scratch file holding it was removed. It appears in no - committed file, log or report. - -## 10. `OPEN-ITEMS.md` rows opened / closed / re-ranked - -- **R-168 → SHIPPED**, with the evidence in the row: the runner, the one-registration-four-repos - measurement, the two probes that changed the design (P2, P6), and the demonstrated alarm with its - accepted-id. -- **R-29 → CLOSED**, both halves — and **closed on the demonstrated alarm, not on a green run**, as - the task required. The class it opened (*a gate that exists, is red, and is invoked by nothing*) is - answered at both ends. -- **R-161 → updated**: its automatic half now exists for the static gate. Its *original* scope, the - runtime volume-persistence gate, is deliberately still **not** automatic and should stay that way. -- **R-169 → NEW.** `grep -oE "R-[0-9]+" | sort -u -t- -k2 -n | tail` established **R-168 as the - highest in use**, so R-169 was free. *CI can only report, because there is no gate in the road* — - making it blocking needs branch protection plus a PR workflow, both of which change how the - operator works. **WAITING-ON-OPERATOR**, owner: operator. The row states the cost honestly (for a - single-operator project a PR workflow may be worse than the disease) and names the real gap: the - window between a `--no-verify` push landing and the operator reading the alarm. -- `ROADMAP.md`: R-168 collapsed to its one-liner, R-29 closed, R-169 added. -- **`architecture/00-capability-map.md` — N/A, stated rather than silently skipped.** No - customer-visible or platform capability changed; this is developer tooling. - -## 11. Was `--no-verify` used? - -**Yes — three times, all on `felhom.eu`, all deliberate and all disclosed.** - -| Commit | Why | +| File | Change | |---|---| -| `3252d51` | **Scenario B itself.** The commit had to carry a broken gate, and pushing it with `--no-verify` *is* the bypass CI exists to catch. The local hook would have refused it — that is the point. | -| `dd13f63` | The tree was still deliberately broken; pushing the alarm step required bypassing the hook that was correctly objecting. | -| `f7dbc33`, `9530de7` | Same, iterating on the alarm while the tree was still red on purpose. | +| `documentation/backlog/OPEN-ITEMS.md` | **R-158** closed (by R-167 — *no second row for the same wire*); **R-167** closed; **R-165** updated with M1-M5 + the operator question, stays open; **4 new rows** R-174/175/176/177 | +| `documentation/backlog/ROADMAP.md` | R-158 collapsed to a shipped one-liner; R-167 added as shipped; R-165 added as spiked/waiting-on-operator | +| `documentation/architecture/00-capability-map.md` | **two new rows**, both **PROVEN-LIVE** with live citations | +| `documentation/architecture/07-backup-architecture.md` §7.5 | **S-1: the contract changed in the same session.** The section's closing claim *"nothing warns when an app crosses the line"* is now false; the alerting is written in, and the one-box-vs-fleet caveat added | +| `CONTEXT.md` | **S-11** (D-c's routing, and why R-158's own `backup_failed` proposal was overruled) and **S-12** (the monitoring landed *before* the merge, not with it) | +| `STATUS.md` | new plain-language section; the merge decision added to *Waiting on you*; **two older entries trimmed** so the page did not grow — one screen, per its own rule | +| `REUSE.md` | the "new event type" extension point rewritten (see §2) | -Every other push in this session ran its pre-push hook and passed. The breakage was reverted in -`4cc1238`, and the hook passed on that push. +## 5. Register IDs -## 12. Observations — noticed, recorded, NOT acted on +**Opened:** R-174, R-175, R-176, R-177. Each established free by +`grep -ro "R-17n\b" documentation/ *.md` → **0 hits**, run before minting. +**Closed:** R-158, R-167, R-174. **Updated, still open:** R-165, R-163 (unchanged — it stays the +record of the constraint until the merge lands). -1. **CI and the pre-push hook never disagreed.** §12 said a disagreement would outrank the CI work; - there was none. The controller's and agent's reuse tallies in CI match the local run exactly. The - one place they *would* have disagreed was designed out rather than discovered: without a sibling - `felhom.eu` clone, both entry points fail closed with `gate is MISSING`. -2. **A near-miss that outranks a probe result.** My first census query reported Actions enabled on - five *unrelated* repos — a baseline drift big enough to change the task. It was a `| tail -5` - inside my own helper truncating the output. The measurement was never wrong; the instrument was. - This is the same class as the `go test -run` filter from session 1, and it is now a `CLAUDE.md` - rule: **an instrument that can silently drop results is not a measurement.** -3. **Two failures on the way to the alarm, both worth their write-up** — `curl: command not found` - (the image is minimal on purpose; fixed with `urllib`, not a bigger image) and **Cloudflare 403 - error 1010** on `api.resend.com`, which blocks the default `Python-urllib` User-Agent. The second - matters because **it looks exactly like an auth failure**: the next person to see a 403 from - Resend should check the User-Agent before rotating a key. Both are in the spike doc and in - `CONTEXT.md` S-9. -4. **A full clone is slow enough to notice** — the P3 probe's `git clone` of `felhom.eu` took ~48 s. - The shipped workflows use `git fetch --depth 1 ` instead, which is both faster and pinned to - the exact pushed commit. Not a finding, but the reason the shipped form differs from the probe's. -5. **Not acted on, out of scope:** Gitea's own mailer is configured and enabled but was never proven - to deliver (§2). If it *is* broken, nothing else in Gitea that relies on email works either — - worth a five-minute test email from the admin UI at some point. It does not affect the CI alarm, - which does not use it. -6. **Not acted on, out of scope:** `homelab-manifests` has no gate entry point and no pre-push hook, - unlike the four product repos, and now no CI workflow either. It is the repo that deploys the - cluster, so that asymmetry may deserve a look — but adding one was not in this task, and the repo - has no gates to run yet. +## 6. CI — run ids and conclusions + +| Repo | Final commit | Run | Conclusion | +|---|---|---|---| +| `felhom-controller` | `9a3c485` | see below | see below | +| `felhom.eu` | (docs commit) | see below | see below | + +*(Filled in from `…/actions/tasks?limit=3` after the final push — see the closing section of the +session output; a green CI that was not looked at is an assumption, not an observation.)* + +## 7. `--no-verify` + +**Not used anywhere.** Every push in this session ran `.githooks/pre-push` (`repo_gates.py --fast` / +`controller_gates.py --fast`) and passed. diff --git a/STATUS.md b/STATUS.md index 2c44eeb..b266943 100644 --- a/STATUS.md +++ b/STATUS.md @@ -47,11 +47,30 @@ change, not assumed. The one thing it still cannot do is *stop* the change: ever straight to the main copy with no review step, so there is no point in the road for it to stand at. It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)* +**A filling disk now warns the customer before anything breaks, and a failed backup now reaches you.** +Until today the first sign that a disk was filling up was a backup that did not happen — nothing said +anything beforehand. Two things changed. The customer is now warned while there is still room to act, +naming the drive and how much space is left, in plain Hungarian that says what to do about it. And +when one app's backup fails for any reason, **you** are told which app and why, with the disk figures +attached — the page that answers "is this app backed up?" was, until now, the one page that never +said. The customer is deliberately *not* told about that second one: they can free up space, but they +can do nothing about a backup that failed, so telling them would only alarm them. + +Both were proven on the demo machine by actually filling a disk. One detail is worth knowing because +it is why there are two rules and not one: the serious warning fired when free space dropped below a +fixed amount while the disk was only 91% full — a percentage on its own would have missed it. + +**These went in *before* the partition change deliberately.** The partition being removed is also a +barrier against a runaway backup filling the space the machine needs to run; putting the warnings in +first means that when it comes down, the thing watching is already working and already tested. +*(R-167, R-158)* + ## What we're working on - **Now:** the last app whose data was never saved; today's decisions written down. -- **Next:** merging the small backup partition into the large one, with the drive-filling warning and - the backup-failure alert in the same step. +- **Next:** merging the small backup partition into the large one — **the warnings for it are already + done and working**, so this step is now only the partition change. It needs one decision from you + first (below). - **After:** rebuilding how the machine records whether an app is meant to be running. ## Waiting on you @@ -63,6 +82,13 @@ It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)* settle both. *(R-110, R-115)* - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a session log; nothing suggests anyone else saw it. *(R-132)* +- **The partition merge: one decision, now measured.** Removing the backup partition also removes a + barrier — today a runaway backup is refused on its own and cannot touch the space the machine needs + to run; afterwards it can, and a machine out of that space is stopped, not slow. So: do we add a + hard stop that refuses a backup before it eats the last of the room, or do we rely on the new + warnings? The recommendation is the hard stop, because it keeps exactly what the barrier gave us. + **Second question, which only you can answer:** does the tester's box need converting in place, or + can it be reinstalled? It does not report to the hub, so nothing here can tell. *(R-165, R-176)* ## Changed since last update @@ -75,25 +101,20 @@ It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)* live. **Also found while checking it: the hub's own database is not in any automatic backup** — it holds every machine's emergency password and the escrow records. Filed, not yet fixed. -- **2026-08-02** — Boot recovery finished. Both halves of the power-cut problem are closed: the - machine records what the customer asked for, and it now waits for the system to finish starting - before deciding what is missing. Six hard resets in a row, everything back every time. A hole the - previous day's change had opened — the machine would have tried to start an app whose external - drive was missing — was found by reading the code, reproduced on the demo box **before** anything - was changed, and fixed the same day. No data ever reached the wrong disk, but only because of a - filesystem accident nothing was relying on deliberately, which is why it was fixed rather than - noted. +- **2026-08-02** — Boot recovery finished: the machine records what the customer asked for, and waits + for the system to finish starting before deciding what is missing. Six hard resets, everything back + every time. A hole the previous day's change had opened — starting an app whose external drive was + missing — was found by reading the code, reproduced on the demo box first, and fixed the same day. + **A second instance of the same hole was found and fixed today**, on the path that restarts an app + after an interrupted backup. -- **2026-08-02** — Fixed and then doubled: thirteen mechanical checks had built up across the four - repositories and nothing ran most of them; two were failing quietly, one since 14 July. Both fixed. - Every repository now has one command that runs all its checks, it runs automatically before every - push, and the build server re-runs it after every push and **emails you when it fails** — proven - with a real broken change. One of the two failures was the setup page claiming the installer was - version 1.19.0 while machines got 1.22.0; that number is now **gone** rather than corrected, - because the page cannot know it — the installer is fetched fresh each run, so any number there is a - guess. A check refuses to let one be added back. +- **2026-08-02** — Thirteen mechanical checks had built up across the four repositories and nothing + ran most of them; two were failing quietly, one since 14 July. Both fixed, and the arrangement that + replaced them is described above. - **2026-08-02** — Decided: the 20 GB backup partition goes away and shares space with app data. That - changes the disk layout, so it happens before any machine is installed outside the house. + changes the disk layout, so it happens before any machine is installed outside the house. **Measured + since:** no machine outside the house is registered yet, so this is as cheap now as it will ever be; + and the two demo machines can simply be reinstalled rather than converted. - **2026-08-02** — Decided: only this machine and the tester's box are protected; every other box, demo boxes included, may be broken or reinstalled freely. Two of the three apps that never saved their data are fixed; this page created. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index f7ceec3..78f6eef 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -84,6 +84,8 @@ | Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) | | An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 | | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped | +| **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait | +| **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it | | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | | | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise | diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 53bb05d..33b2469 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -529,6 +529,28 @@ portable secrets in the unit. **That independence is bounded by app size**, and > Past that the unit cannot be captured, and the app falls back to Lane 2's operator-driven > whole-guest route. +**AS OF 2026-08-02 SOMETHING NOW WARNS, AND THE ALERTING IS PART OF THIS CONTRACT (R-167 / R-158, +decision D-c; controller v0.191.x + hub v0.89.0).** The last sentence of this section used to end +"nothing warns when an app crosses the line". Two signals now exist and both are PROVEN-LIVE: + +- **To the CUSTOMER, before anything fails** — `internal/fillwatch` warns per FILESYSTEM (never per + app: one full disk holding ten apps would fire ten times) on **whichever trips first, used ≥ 85% or + free < 5 GiB**, critical at 95% / 2 GiB, clearing at 75% / 7 GiB. **Two terms, because a percentage + alone lies at both ends of the range this section itself documents:** 85% of a 20 G `mp1` leaves + 3 G — less than one DB-backed app's unit — while 85% of a 4 TB drive leaves 600 G. It watches the + app-data volume, the system-data volume **and** every registered drive, which the previous + `health_degraded` signal did not. Edge-triggered against persisted state; the hub owns cooldown. +- **To the OPERATOR, when a capture actually fails** — `recovery_unit_capture_failed`, per app, with + the target filesystem's used/free bytes at the moment of failure, so the *why* needs no login. It is + **operator-tier** (`notify.operatorOnlyEvents`) and deliberately not `backup_failed`: a customer can + take no action on a capture failure. + +**A caveat this section must carry, because the bound below depends on it.** The `mp0 50G / mp1 20G` +table above is **demo-hp's** shape, not the fleet's — demo-felhom ships `mp0 200G / mp1 50G`, where +the same bound is ≈ 49 GB / ≈ 24 GB, and the golden's own defaults are `16 G / 8 G` before provision +grows them. **The bound below is a FUNCTION of `mp1`, not a constant.** Measured 2026-08-02, +`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1; correcting the numbers throughout is **R-175**. + Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling** (**R-163**) — this section states the constraint, not a number. And **the same-device placement is intended, not a defect**: a driveless app's unit sits on the same SSD as its volumes, but Tier-2's diff --git a/documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md b/documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md new file mode 100644 index 0000000..5dc0a52 --- /dev/null +++ b/documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md @@ -0,0 +1,248 @@ +# SPIKE — measuring the `mp1` → `mp0` merge (R-165, decision D-a) + +**Date:** 2026-08-02 · **Author:** Claude Code · **Status:** MEASUREMENT ONLY — **no layout changed** + +> **Decision D-a is already taken** (`CONTEXT.md` S-5): the dedicated backup partition is merged away +> rather than resized. This document measures **how**, never **whether**. It ends in a question for the +> operator and a **STOP** — the merge itself is next session's supervised work. + +## 0. What was and was not touched + +Nothing was created, resized, moved or deleted. No golden was rebuilt, no guest config edited, no +partition altered. Every figure below is a read of live state or of committed source. `ep0` and Peti's +box were not contacted at all (`runbooks/target-selection.md`, decision D-d). + +The monitoring D-a requires (**R-167**) shipped **before** this measurement, in controller v0.191.x and +hub v0.89.0, and was proven on guest 9201. That ordering is the point: D-a's condition (2) says the +monitoring lands with the merge and never after, and landing it first is strictly better. + +--- + +## M1 — what is actually there + +**`pct config 9201`, both demo hosts, read 2026-08-02.** + +| | demo-felhom (N100) | demo-hp (t740) | golden default | +|---|---|---|---| +| `rootfs` | 32 G | 32 G | `OS_SIZE_GB=32` | +| `mp0` `/var/lib/docker` | **200 G**, `backup=1` | **50 G**, `backup=1` | `GOLDEN_DOCKER_GB=16` | +| `mp1` `/mnt/sys_drive` | **50 G**, `backup=1` | **20 G**, `backup=1` | `GOLDEN_SYSDATA_GB=8` | +| `mp8` `/mnt/felhom-drives` | bind | bind | — | +| `mp9` bootstrap | bind, `ro=1` | bind, `ro=1` | — | + +**Occupancy (`df`, in-guest):** + +| filesystem | demo-felhom | demo-hp | +|---|---|---| +| `/` | 945 M / 32 G (4%) | 942 M / 32 G (4%) | +| `/var/lib/docker` (`mp0`) | 13 G / 197 G (7%) | 5.4 G / 50 G (12%) | +| `/mnt/sys_drive` (`mp1`) | **2.0 G / 50 G (5%)** | **92 M / 20 G (1%)** | +| data drive | 21 G / 916 G (3%) — `hdd_1` | 6.7 G / 938 G (1%) — `nvme-1tb` | + +**THE FIRST FINDING IS THAT "THE LAYOUT" IS NOT ONE THING, AND THE SPREAD IS WIDER THAN §7.5 RECORDS.** +`architecture/07-backup-architecture.md` §7.5 states the default appliance as `mp0 50G / mp1 20G` — +that is demo-hp exactly, and it is **not** demo-felhom, which ships **200 G / 50 G**, four and two and +a half times larger. Both differ again from the golden's own `16 G / 8 G`, because `provision` grows +the volumes after restore. **Any merge plan expressed as a fixed pair of numbers is already wrong for +one of the two boxes that exist.** + +**The corollary matters more than the numbers.** §7.5's headline bound — *"≈ 19 GB of app data for a +file-only app, ≈ 10 GB for a DB-backed one"* — is derived from `mp1 = 20 G`. On demo-felhom, where +`mp1` is 50 G, the real bound is ~49 GB / ~24 GB. **The architecture doc states one bound as if it +were the fleet's, and it is one box's.** That is a documentation defect independent of the merge and +is filed as its own row. + +--- + +## M2 — what lives on `mp1` (it is not only backups) + +**`du`, in-guest, read 2026-08-02.** + +``` +demo-felhom /mnt/sys_drive/felhom-data 2.0 G + ├── backups/primary 269 M Tier-1 units of DRIVELESS apps (~30 apps) + ├── backups/secondary 1.7 G Tier-2 mirrors + └── userdata/import 12 K the canonical drop-zone (R-75) + +demo-hp /mnt/sys_drive/felhom-data 92 M + ├── backups/primary uptime-kuma, paperless + ├── backups/secondary paperless-ngx + └── userdata/import paperless +``` + +**Four distinct things would move, not one**, confirming the task's warning that a plan accounting +only for the units is wrong: + +1. **`backups/primary/`** — the RETAINED Tier-1 recovery unit of every app with no data drive. + Thirty apps on demo-felhom. +2. **`backups/secondary/`** — Tier-2 cross-drive mirrors, including the `_shares` pseudo-stack. + On demo-felhom this is **1.7 G of the 2.0 G — the majority is Tier 2, not Tier 1.** +3. **`userdata/import`** — the canonical, app-INDEPENDENT drop zone (`GetImportRoot`, R-75). It lives + on the system drive **by contract**, not by convenience. +4. **`userdata/`** more generally — the system-data userdata namespace for driveless apps. + +**Observed occupancy is far below capacity on both boxes** (5% and 1%). The measured pressure today is +zero; the constraint R-163 records is a *ceiling* problem, not a *current fill* problem. That is worth +stating plainly because it bears on urgency, not on correctness. + +--- + +## M3 — which merge shapes exist, and what each breaks + +Three interacting assertions exist today, and a merge touches all three. **All three were read at +source; none had been measured before.** + +**(a) The golden build ASSERTS the split and ABORTS if it is absent.** +`felhom-agent/configs/build-golden.sh:130`: + +``` +findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . || { + echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; } +``` + +There is a sibling assertion for `mp0` at :126, and **two more** at :315-324 that abort if the vzdump +log shows `excluding volume mount point mp0` or `mp1`. So the golden build fails closed on the split +in **four** places, not one. + +**(b) The whole-guest archive's scope is `rootfs + mp0 + mp1`,** and only because both carry +`backup=1`. `build-golden.sh:69` records why: *"backup=1 is MANDATORY: without it vzdump EXCLUDES the +volume (extra mountpoints default backup=0)"*. + +**(c) `mount_parity` compares the ARCHIVE's `mpN` set against the RESTORED guest's** +(`felhom-agent/internal/reconcile/restoretest.go:271-283`, `mountParity` at :347). Per slot it requires +the same mount path and a restored size **not smaller** than the archive's, and it checks the reverse +direction too. A mismatch fails the restore-test outright. + +| shape | (a) golden assertion | (b) archive scope | (c) `mount_parity` | verdict | +|---|---|---|---|---| +| **S1 — one volume, two directories.** `mp1` stops existing; `/mnt/sys_drive` becomes a directory on the `mp0` filesystem | **BREAKS** — :130 aborts the build. Must be deleted, and :319's vzdump guard for `mp1` with it | **HOLDS** — scope becomes `rootfs + mp0`, still complete, because the data moved onto `mp0` | **HOLDS for new archives** (no `mp1` in archive ⇒ none required in restore). **A pre-merge archive restored into a merged guest is a different question — see below** | the shape D-a describes | +| **S2 — two mounts, one backing pool.** `mp0` and `mp1` remain separate `mpN` slots on the same storage | **HOLDS** — both are still separate mounts | **HOLDS** unchanged | **HOLDS** unchanged | **does NOT remove the ceiling** — two filesystems still have two independent `df`s. This is thin-provisioning, not a merge, and it converts a clean per-app refusal into a shared-pool exhaustion that neither volume can see coming | +| **S3 — grow `mp1`, keep the split** (**the shape D-a REJECTED — measured here as the baseline**) | **HOLDS** | **HOLDS** | **HOLDS** | zero structural risk, one `--sysdata-grow` value. It is *"the same wall further away"* — D-a's own words — and does not close R-163 | + +**The one genuinely unmeasured item in M3, stated as unmeasured:** whether a **pre-merge archive** +(carrying `mp1`) restores cleanly into the merged world. Reading `mountParity` says it should — the +restore recreates `mp1` from the archive, so archive and restored guest agree, and parity passes. But +**that was reasoned from source and not executed**, and this project's own record is that four +production designs specced against unvalidated mechanisms were all wrong. **It is a one-command +restore-test on a Tier-0 box and should be run before the merge, not after.** + +--- + +## M4 — the bulkhead question (the important one) + +**Today `mp1` is not only a ceiling; it is a BULKHEAD.** An app whose recovery unit outgrows the space +is refused **per app**, its last good unit is preserved **byte-identical** (R-158's measurement), and +— critically — **the overflow cannot reach `/var/lib/docker`**, because it is a different filesystem. +The container runtime keeps running. + +After S1 the same overflow lands on the filesystem Docker itself runs on. A runaway recovery-unit +capture would fill `/var/lib/docker`, and a full Docker data-root is not a degraded state, it is a +stopped one. + +**This is the one place where "the merge is cheap" stops being true**, and it is why the warnings +shipped first rather than alongside. + +Four candidate replacements. **No choice is made here — this is the operator's ruling.** + +| # | replacement | what it costs | what it leaves open | +|---|---|---|---| +| **B1 — a reserved block percentage on the merged filesystem.** `tune2fs -m` reserving N% for root; the controller runs unprivileged, so a capture cannot consume the reserve while root-owned Docker can | one `tune2fs` at build time; no code | the reserve protects *root*, not *Docker's runtime need* specifically; sizing it is a guess without a measured worst case | +| **B2 — a refusal threshold in the capture path.** `captureAllRecoveryUnits` refuses when free space would drop below a floor, per app, and emits the alert R-158 just wired | small, local, testable; **reuses the alert that now exists** and restores the per-app-refusal semantics the bulkhead gave for free | the floor is a number needing justification; it protects the capture path only — a customer filling `mp0` through app data is untouched by it | +| **B3 — a filesystem quota on the backup directory.** XFS project quota / ext4 project quota on `/sys_drive/backups` | enforces at the filesystem, so **every** writer is bounded, not only the capture path | recreates a fixed ceiling — i.e. it is R-163 again inside one volume, and D-a's objection to *"a bigger number is the same wall further away"* applies to it word for word | +| **B4 — R-167's warnings are deemed sufficient.** No hard stop; the customer is warned at 85% / 5 GiB and critically at 95% / 2 GiB | zero — it is already shipped and proven live | **a warning is not a bulkhead.** It depends on a human acting within the window, and the failure it fails to prevent is "Docker's data-root is full", which is the worst failure on the box | + +**The measured input to that ruling:** on demo-felhom `mp1` holds 2.0 G against a 50 G ceiling, and +`mp0` has 175 G free. The overflow scenario is not close today on either box. **B2 is the only option +that preserves the property the bulkhead actually provided** — a per-app refusal with the last good +unit intact — and it is the one that reuses what R-158 just built. + +--- + +## M5 — existing boxes, and what a migration costs + +**Read from the hub's own registers, 2026-08-02. No box was contacted.** + +The hub's `/hosts` register holds **four** hosts, of which **two are ONLINE**: + +| host | customer | agent | status | +|---|---|---|---| +| `demo-felhom-8363b5` | Demo Ügyfél | 0.119.0 | **ONLINE** | +| `demo-hp-bb76ea` | Demo HP | 0.119.0 | **ONLINE** | +| `drill-r50-0a4f9a` | drill-r50 | 0.113.0 | DOWN | +| `sess-f-2670b5` | R-120 golden 0.186.0 proof | 0.116.0 | DOWN | + +**The customer register lists five customers** — `david`, `demo-felhom`, `demo-hp`, `peti-felhom`, +`sess-f`. + +**THE FINDING THAT CHANGES THE COST: `peti-felhom` EXISTS AS A CUSTOMER BUT HAS NO HOST IN THE +REGISTER.** Consistent with the long-standing *"guest not on agent node"* stop in the Peti-return +runbook. So: + +- **The two demo boxes are the entire measurable migrated population, and both are Tier 0 — + disposable, per decision D-d.** For them a "migration" is not required at all: they can be + **reinstalled** from a merged golden, which is cheaper and lower-risk than migrating in place, and + D-d explicitly permits it. +- **D-a's condition (1) — "it must land before any external install" — is currently SATISFIED.** No + external box appears in the hub's host register. **This is the cheapest this decision will ever be, + and the window is open now.** + +**UNMEASURED, and reported as unmeasured rather than omitted:** Peti's box's actual disk layout. It +does not report to the hub, so the hub holds no `pct config` for it, and it is protected by D-d and +`target-selection.md` — so it was not contacted. **Whether a merged golden implies an in-place +migration for that box, and what that costs, is not established by this spike.** Nor is whether the +box is currently restorable at every point of such a migration. Both are inputs the operator has and +this session does not. + +**Also unmeasured:** the per-box in-place migration procedure itself (move `/felhom-data` onto +`mp0`, drop the `mp1` slot, verify) has not been executed even once on a throwaway guest. If an +in-place migration is ever needed, **that rehearsal is the first thing to do, and its "is the box +restorable at every point?" answer is currently unknown.** + +--- + +## Ranked options, and a recommendation + +**Ranked by what the measurements support, not by preference.** + +1. **S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), shipped as a + FRESH-INSTALL shape, with the demo boxes REINSTALLED rather than migrated.** + It is what D-a describes; it genuinely removes the ceiling rather than moving it; the four golden + assertions and the two vzdump guards are a bounded, greppable edit; the archive scope stays + complete; and B2 restores the per-app refusal that is the bulkhead's real value, reusing the alert + R-158 just wired. The migration cost for the measurable population is **zero**, because both boxes + are Tier 0 and reinstallable. +2. **S1 + B4 (warnings only).** Cheapest, and everything it needs is already shipped and proven live. + Rejected as the *recommendation* only because the failure it declines to prevent — a full Docker + data-root — is the worst one on the box, and it depends on a human acting inside the window. +3. **S3 (grow `mp1`, keep the split).** Zero structural risk, one number. It is the measured baseline + and D-a rejected it; recorded so the decision is compared against something. +4. **S2 (two mounts, one pool).** Not recommended at all. It keeps both assertions satisfied while + delivering none of the benefit, and converts a clean per-app refusal into a shared-pool exhaustion + neither `df` can see coming — strictly worse than today. + +**Two things to do BEFORE the merge session, both cheap, both currently unmeasured:** + +- **Run one restore-test of a PRE-MERGE archive into a merged-layout guest on a Tier-0 box.** M3 + reasons it passes; nothing has executed it. This project's own record is that reasoning from source + about an unvalidated mechanism has been wrong four times. +- **Rehearse the in-place migration once on a throwaway guest**, and record whether the guest is + restorable at every point of it. Only needed if Peti's box turns out to require migrating rather + than reinstalling — which is the operator's information, not the hub's. + +--- + +## THE QUESTION FOR THE OPERATOR + +**Which shape, and what replaces the bulkhead?** + +The recommendation is **S1 + B2**, fresh-install shape, demo boxes reinstalled. The two open inputs +only the operator has are: + +1. **Does Peti's box need an in-place migration, or can it be reinstalled?** The hub cannot answer + this — the host is not in its register. It decides whether the migration rehearsal is required work + or optional insurance. +2. **Is B2's per-app refusal the right replacement for the bulkhead, or is B4 (warnings only) + acceptable?** B2 costs a small amount of code and a justified floor; B4 costs nothing and is + already live, but accepts that a full Docker data-root is reachable. + +**STOP.** The merge is next session's work, and it is a supervised one. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index db455fe..dadda18 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -89,17 +89,21 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-170** | ~~**The drive-backed boot gate infers a customer's Stop from a container count.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | `shouldRecreateOnBoot` now reads `desired_state` with the SAME three-way table as `isBootOrphan`: `stopped` → never; `running` → recreate whatever the container count; **absent → exactly the pre-v0.190.0 `hasContainers` behaviour**. `presentStable` untouched and still load-bearing (an absent drive is never recreated here — the very term the boot sweep was missing, R-171). Its comment argued at length FOR the container count and was rewritten; a correct implementation under a comment arguing the opposite is worse than either alone. **The agreement is pinned from BOTH sides** against one fixture table (`TestBothBootGatesAgreeOnIntent` / `TestShouldRecreateOnBoot_AgreesWithBootrecon`) because the two gates cannot be called from one package without an import cycle. **Live on 9201, both halves in one reboot:** calibre-web (drive-backed, `running`, ZERO containers) → `recreating drive-backed app calibre-web`; immich (`stopped`) → `1 drive-backed app(s) left stopped on purpose` | — | | **R-171** | **The boot sweep started apps whose data drive was ABSENT — a regression introduced by v0.189.0, now FIXED.** Replacing `isBootOrphan`'s container-count term with recorded intent made a drive-gate-stopped app (`compose down` ⇒ zero containers, and the gate never touches `desired_state` because it is not the customer) read as a boot orphan | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | **Reasoned from the diff, then CONFIRMED on hardware before any fix was written** (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The sweep found and started calibre-web with its drive unmounted, burned both attempts and handed it to the dead-app alarm — **a false alarm about an app the drive gate is deliberately holding**. The *write* hazard did NOT materialise: compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is host-root-owned and the guest is unprivileged — **an accidental protection no code owns, no test pins, and one `chown` or one privileged guest away from gone**. Fix: new consumer-side seam `bootrecon.StartGate`, **fail-safe (cannot determine ⇒ do not start)**, wired in `main.go`; `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two cannot drift. **The rule is not new** — the API's `startGatedByMissingDrive` already refused this to the customer; the sweep bypassed it by calling `Manager.StartStack` directly. Widening the window (R-157 A) made two more holders reachable, so the same seam also refuses an app held by a **quiesce** or an **in-flight app-data operation** (§8.2), reusing `quiesce.SuppressedStacks()` and a new read-only `AppStopGuard.HeldStacks()`. Held apps report as `HeldByDrive`, never `StillDown` — that is the alarm's bucket. **ID established free:** `grep -ro "R-171\b" documentation/ *.md` → 0 hits before minting | — | | **R-172** | ~~**A false `host_stale` alarm fires when the hub's SQLite refuses two consecutive host reports.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (hub v0.88.0, 2026-08-02) | — | **ROOT CAUSE WAS NOT TUNING — THE PRAGMAS WERE NEVER APPLIED.** `store.New` used `?_journal_mode=WAL&_busy_timeout=5000`, which is **mattn/go-sqlite3** syntax; the driver is **modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`/`_time_format`/`_time_integer_format`/`_txlock`/`_inttotime` and **ignores the rest without an error**. The hub ran in rollback-journal mode with `busy_timeout=0` for its entire life while its own source said WAL — a configuration asserting an invariant the code did not provide. Proof: a 128 MB open `/data/hub.db` with **no `-wal`/`-shm` beside it**. **Fix:** `?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate`. **`_txlock=immediate` is not optional** — `database/sql`'s `Begin()` is DEFERRED, so a read-then-write tx must upgrade its lock and a failed upgrade is `SQLITE_BUSY_SNAPSHOT`, which **`busy_timeout` does not retry**; this store has 10+ `db.Begin()` sites, all write paths. **Retry options (b) and (c) were deliberately NOT taken** — with readers no longer blocking writers a surviving `SQLITE_BUSY` would be a real signal, and a retry would hide it; revisit only on evidence. **Live:** `-wal`+`-shm` now present, **zero `SQLITE_BUSY` since rollout**, host back to `ok`, and `PRAGMA integrity_check` = `ok` with `journal_mode=wal` after three unrelated OOM restarts. **Operational consequence handled:** a WAL DB cannot be copied by taking `hub.db` alone — the break-glass retrieval in `operations/nodes.md` did exactly that and is now WAL-aware (the live `-wal` was 729 KB, i.e. a bare `cat` would have silently omitted it) | — | +| **R-174** | ~~**The app-stop guard's crash recovery started apps onto MISSING drives — a regression in v0.189.0 code.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0, 2026-08-02) | — | **Found by REVIEW on 2026-08-02, in code shipped 2026-08-01, and closed the same session — R-171 one path over.** `appStopGuard.SetStarter(stackMgr)` handed `Recover` the RAW stack manager, whose `StartStack` has no drive gate, and `Recover` runs **at startup** — exactly when an external drive may not have come back. So: a backup stops an app, the box loses power, the drive does not remount, and the app is started on a missing drive. The rule was not new — the API's own `startGatedByMissingDrive` already refused this to the customer; the guard bypassed it. **`bootDriveGate` could NOT be reused whole**, and the reason is recorded in the code: its holder #2 reads `bootAppStopGuard.HeldStacks()`, which during `Recover` is **the guard's own marker** — it would refuse every recovery it was meant to perform — and holders #1/#2 read package-level vars assigned AFTER `Recover()` runs, so a whole-gate reuse would be correct only by accident of nil-safety. Holder #3 is extracted into a shared `driveStartGate` with **two callers, one implementation**, and `TestBootDriveGateAndAppStopShareTheDrivePredicate` pins the delegation. **A REFUSAL IS NOT A FAILURE:** new `ErrStartRefused` + a `Refused` bucket — both keep the marker, only `Failed` alarms, because routing a deliberate hold into `NotifyBackupFailed` (customer-enabled by default) is the very R-171 false alarm this fixes. `main.go` guards on `Alarming()`, not `!= nil`, and the pre-existing seam test was TIGHTENED to require it. **Live on 9201, both directions:** drive held unmounted → `refusing to restart "calibre-web" … drive /mnt/felhom-drives/hdd_1 is not a live mountpoint`, marker retained byte-identical, zero containers started, `not alarming`; drive returned → `restarted calibre-web`, marker CLEARED. **ID established free:** `grep -ro "R-174\b" documentation/ *.md` → 0 hits | — | +| **R-175** | **`07-backup-architecture.md` §7.5 states ONE box's size bound as if it were the fleet's.** Its headline — *"an app can be restored by Lane 1 only while its recovery unit still fits the retained space — ≈ 19 GB of app data for a file-only app, ≈ 10 GB for a DB-backed one"* — is derived from `mp1 = 20 G`. **That is demo-hp exactly, and it is not demo-felhom**, which ships `mp0 200G / mp1 50G`, where the real bound is ≈ 49 GB / ≈ 24 GB. The golden's own defaults are a third pair (`GOLDEN_DOCKER_GB=16` / `GOLDEN_SYSDATA_GB=8`), grown at provision time | **READY (S) — NEW 2026-08-02** | — | **Measured, not inferred** (`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1: `pct config 9201` on both hosts). Independent of the merge — the sentence is wrong today and will be wrong differently after R-165. **The fix is to state the bound as a FUNCTION of `mp1`, not a constant**, and to say which box any quoted figure came from. Same class as the comment-asserting-an-invariant rule: a doc stating a fleet-wide number that only one machine satisfies reads as settled and is not. **ID established free:** `grep -ro "R-175\b" documentation/ *.md` → 0 hits | CC | +| **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **READY (S) — NEW 2026-08-02** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits | CC | +| **R-177** | **There is no operator-triggerable "run the fill check now" path.** `fill-watch` is reachable only on its daily 03:30 schedule plus the once-at-startup run added in controller v0.191.1 — so the only way to exercise it on demand is to restart the controller | **READY (S) — NEW 2026-08-02** | — | **Noticed while live-validating R-167 on 9201, not by a failure.** It cost a controller restart per observation during validation, and it costs the same on a support call: after a customer frees space, nobody can confirm the warning has cleared without restarting their controller or waiting until 03:30. **Partially mitigated already** — v0.191.2 makes every run log a positive observable (`checked N filesystem(s), M unreadable/skipped, K notification(s)`), so at least a run that DID happen is visible; the gap is triggering one. The scheduler has `GetJobs` but no run-now, so this is a general affordance, not a fill-watch one — **scope it as "run a named scheduler job now", operator-gated.** **ID established free:** `grep -ro "R-177\b" documentation/ *.md` → 0 hits | CC | | **R-173** | **The hub's SQLite PVC is excluded from every Longhorn backup job.** `pvc/hub-data` carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly` (04:00 / Sun 05:00) are the ONLY recurring jobs and both target the `default` group — so the 128 MB `/data/hub.db` has **no volume-level backup**. That database holds `host_recovery` (every managed box's break-glass root password), `host_escrow` + `host_escrow_superseded` (escrow custody), `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg endpoints/peers — i.e. the material several documented recovery routes depend on | **READY (M) — NEW 2026-08-02** | — | **Noticed while checking the blast radius of the R-172 WAL change, not by a failure** — the WAL work needed to know who copies this file, and the answer turned out to be nobody on a schedule. **Establish before designing:** (a) whether the exclusion is deliberate (a 1 Gi RWO Longhorn volume snapshotting a 128 MB SQLite file is cheap, so the label looks like a leftover rather than a decision) and by whom; (b) whether anything else backs it up out-of-band that this census missed — the `_recovery-inventory-2026-07-28.md` records a MANUAL hot copy, which is not a backup. **When it is designed, it must be WAL-aware** (R-172): a volume snapshot of a live WAL database is crash-consistent and replays on open, which is fine, but any file-level copy must take `hub.db-wal` too or it silently loses the newest writes. **Grep establishing the ID was free:** `grep -ro "R-173\b" documentation/ *.md` → 0 hits | CC | -| **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC | +| **R-158** | ~~**A local Tier-1 app-data backup failure reaches no hub channel.**~~ | **CLOSED BY R-167 — SHIPPED + PROVEN-LIVE** (controller v0.191.0 + hub v0.89.0, 2026-08-02) | — | **Closed by the wire it named; no second row was filed for it** (R-167 subsumes and widens it). New `unitNotify` seam + `SetUnitNotify` beside the manager's existing three, called from `captureAllRecoveryUnits` **per app with the loop continuing**, carrying the target filesystem's used/free bytes at the moment of failure — the cause is usually a full filesystem and those numbers answer *why* without an operator logging in. **ROUTED TO THE OPERATOR, NOT `backup_failed`, AND THAT OVERRIDES THIS ROW'S OWN PROPOSAL.** The proposal above said *"emitting the existing `backup_failed`"*; that type carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents`, so it would email the customer in Hungarian about a failure they cannot act on — precisely the mistake R-97a avoided by minting `whole_guest_backup_failed`. **Decision D-c routes it to the operator and D-c wins.** New `recovery_unit_capture_failed` in `allowedEventTypes` **and** `notify.operatorOnlyEvents`; `notify.IsOperatorOnly` added so ONE test pins both registers (allowlisted-but-not-operator-only is invisible when they are checked separately — the v0.78.0 defect). **Red-proof:** removing the register entry shows the customer being emailed. **Live on 9201:** two events accepted and stored, `operator | sent`, and the positive observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from the hub's `notification_log` | — | | **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `_` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC | | **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC | | **R-161** | **The volume-persistence gate is enforced by CONVENTION, not automatically.** The catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none exists). | **REDUCED SCOPE — open** (operator ruling 2026-08-02) | a second person touching templates | **RULED. Both obvious enforcement points were rejected for measured reasons.** *Controller-side at template load:* rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — **it would pass on the exact defect it exists to catch**; the property is decidable only at runtime. *CI:* rejected for now — neither repo has any, and there are no users yet. **SHIPPED instead** (`app-catalog-felhom.eu` `fd7747d`): `scripts/catalog_gates.py`, ONE entry point running all three gates, non-zero exit on any failure, **mandated in the catalog's `CLAUDE.md`** the way `site_gates.py` is. Rationale for the record: of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md — `site_gates.py` is run, R-29's three orphans are named nowhere and have stopped nothing. **What remains open is only the automatic half:** this is convention, run by a person, and that is sufficient while one person touches templates. Revisit when a second does **UPDATE 2026-08-02:** `catalog_gates.py` gained `--fast` (gate 1 only — the network and runtime gates are deliberately NOT in a hook: a push that pulls images and starts containers gets bypassed within a week and the bypass becomes the habit) and `.githooks/pre-push` now runs it. **The automatic half now has a designated successor row: R-168** (Gitea Actions runner). This row stays open at its reduced scope — the runtime gate remains a deliberate periodic run **UPDATE 2026-08-02 (second):** the automatic half now EXISTS — R-168's runner executes `catalog_gates.py --fast` on every push to this repo (measured: run #1, `image-pin gate OK — 53 templates`, with the two runtime gates announced as skipped and their own output absent from the log). This row's *original* scope — the RUNTIME volume-persistence gate — is deliberately still NOT automatic and should stay that way: CI that pulls 53 images on every push gets disabled. It remains a periodic run | operator | | **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC | | **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio** | — (the sizing question is answered; the work is **R-165**) | **RE-FRAMED, NOT CLOSED (operator decision D-a, 2026-08-02 — `CONTEXT.md` S-5).** The row asked *what ratio should `mp1` be?* and that question is **withdrawn rather than answered**: `mp1` is merged into `mp0` so local recovery units share the app-data area and the ceiling stops existing — a bigger number is the same wall further away. **This row stays open as the record of the constraint** (what `mp1` is for, what it gates, and the measured 2× DB-app unit size) **until the merge lands**, because until then every consequence below is still live on every box. **The work is R-165; the warning that must ship with it is R-167.** Original finding, unchanged, follows. **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 (widened to R-167) | CC | | **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC | -| **R-165** | **Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.** Operator decision **D-a**, 2026-08-02 (`CONTEXT.md` S-5). Local recovery units share the app-data area instead of holding their own fixed ceiling, so the wall R-163 describes is removed rather than moved further away. Guest 9201 on demo-hp ships `mp0 50G / mp1 20G` today | **READY (M) — NEW 2026-08-02** | — | **Two conditions travel WITH the decision and are not optional.** **(1) Before any external install.** It changes the **disk layout**, so it is a fresh-install shape while there are no external boxes and a per-box migration after — and the decision's cheapness is entirely a function of that ordering. **(2) It removes a wall that currently fails safely**, so **R-167** (D-c: fill warning + failure alert) lands in the same step, never after: today an app that outgrows `mp1` is refused per app with the last good unit preserved byte-identical (R-158's measurement), and after the merge the same overflow consumes the space the app itself is using. Touches the installer/agent guest shape (`--sysdata-grow` defaults to **0** and is not derived from the physical drive, `felhom-agent/cmd/felhom-agent/main.go:178`) and the golden. **Does NOT close R-163** — that row is the record of the constraint and stays open until this lands | CC | +| **R-165** | **Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.** Operator decision **D-a**, 2026-08-02 (`CONTEXT.md` S-5). Local recovery units share the app-data area instead of holding their own fixed ceiling, so the wall R-163 describes is removed rather than moved further away. Guest 9201 on demo-hp ships `mp0 50G / mp1 20G` today | **READY (M) — NEW 2026-08-02** | — | **Two conditions travel WITH the decision and are not optional.** **(1) Before any external install.** It changes the **disk layout**, so it is a fresh-install shape while there are no external boxes and a per-box migration after — and the decision's cheapness is entirely a function of that ordering. **(2) It removes a wall that currently fails safely**, so **R-167** (D-c: fill warning + failure alert) lands in the same step, never after: today an app that outgrows `mp1` is refused per app with the last good unit preserved byte-identical (R-158's measurement), and after the merge the same overflow consumes the space the app itself is using. Touches the installer/agent guest shape (`--sysdata-grow` defaults to **0** and is not derived from the physical drive, `felhom-agent/cmd/felhom-agent/main.go:178`) and the golden. **Does NOT close R-163** — that row is the record of the constraint and stays open until this lands **MEASURED 2026-08-02 — `audits/SPIKE-r165-mp1-merge-2026-08-02.md`; no layout was touched.** **M1: "the layout" is not one thing** — demo-felhom ships `mp0 200G / mp1 50G`, demo-hp `50G / 20G`, the golden `16G / 8G`; any plan expressed as a fixed pair is already wrong for one of the two boxes (this also invalidates §7.5's fleet-wide bound → **R-175**). **M2: the majority of `mp1` is NOT Tier-1** — on demo-felhom 1.7 G of 2.0 G is Tier-2 mirrors, plus the `userdata/import` drop zone which is on the system drive by CONTRACT (R-75); a plan accounting only for the units is wrong. Observed fill is 5% / 1% — the constraint is a ceiling problem, not a current-fill one. **M3: three assertions break, and the golden fails closed on the split in FOUR places** (`build-golden.sh:126,130` separate-mount + `:315,319` vzdump-exclusion guards), not one; the archive scope `rootfs+mp0+mp1` stays complete after the merge; `mountParity` holds for new archives. **M4 — THE BULKHEAD IS THE REAL COST:** today an overflow is refused per app with the last good unit byte-identical AND CANNOT REACH `/var/lib/docker`; after the merge it can, and a full Docker data-root is a stopped box, not a degraded one. Four candidate replacements costed (reserve / capture-path refusal / project quota / warnings-only); **not chosen — operator's ruling.** **M5: D-a's condition (1) is currently SATISFIED** — no external box is in the hub's host register (only the two ONLINE demo boxes, both **Tier 0 and reinstallable**, so migration cost for the measurable population is ZERO). **`peti-felhom` exists as a customer with NO host in the register, so its layout is UNMEASURED** and was not contacted (D-d). **Recommendation: S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), as a fresh-install shape with the demo boxes REINSTALLED.** **Two prerequisites are unmeasured → R-176.** **STOPPED for the operator's ruling; the merge is a supervised session.** | CC | | **R-166** | ~~**App state gets a desired/observed model with its own store.**~~ Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5) | **SHIPPED + PROVEN-LIVE** (controller v0.189.0, 2026-08-02) | — | **Both blocking facts were established at source before any code was written, and the answers changed the shape.** **(a) Does a crash-safe journal already exist for the in-flight case?** YES, twice — `internal/quiesce/quiesce.go` (marker + `Recover`, proven on live hardware by Campaign 8 fault 10) and `internal/stacks/migrate.go` (`migration.json` + `RecoverMigration`) — but **neither covers the app-data path**: `DumpAppVolumesSafe` stopped and restarted an app with **no marker, no journal and not even a `defer`**. So the pattern existed and the coverage did not; `backup.AppStopGuard` copies the proven shape into its **own** file (one file, one writer). **(b) Is the SQLite store reachable?** Irrelevant, and deliberately unused: `metrics.db` is optional by design (the controller runs with it absent), and operational state must not live in a store designed to be droppable. **Shipped:** tri-state `desired_state` in `app.yaml` written ONLY by the customer's action (API action switch, `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, `.fab` import — a 14-caller census established that `StartStack`/`StopStack` must NOT be writers); `isBootOrphan` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN, never running**, so a legacy `app.yaml` keeps byte-identical pre-v0.189.0 behaviour; running-only backfill. D-b's every-container requirement was already met by `aggregateState` and was NOT re-implemented. **Live on 9201:** all three flows (stop survives a restart; a zero-container `running` app is recovered by name; a legacy app.yaml is skipped and never inferred as stopped). **Also fixed en route:** `SaveAppConfig` rebuilt `AppConfig` field-by-field (the R-100 shape) and would have dropped the new field on every save across nine call sites | — | -| **R-167** | **Storage monitoring and backup alerts.** Operator decision **D-c**, 2026-08-02 (`CONTEXT.md` S-5). Two pieces: a warning to the **customer** as a drive fills, **before** anything fails; and a failure alert to the **hub/operator** when any backup fails for **any** reason, lack of space included | **READY (M) — NEW 2026-08-02** | — | **Prerequisite for R-165** — D-a removes a wall that currently fails safely, so this ships in the same step, not after. **Subsumes and widens R-158**, which is the same alert one tier down (`NotifyBackupFailed` exists and the hub allowlists `backup_failed`; only the off-box/NAS leg calls it, `cmd/controller/main.go:659`). R-158 stays as the named seam and its proposed `unitNotify` shape is the concrete first leg — **do not file a second row for the same wire.** The customer half is new: nothing today warns before a fill, and R-158's measurement is that when it happens exactly one page says so | CC | +| **R-167** | ~~**Storage monitoring and backup alerts.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02) | — | **Operator decision D-c. It shipped BEFORE the R-165 merge, not with it** — D-a's condition (2) says the monitoring lands in the same step and never after, and landing it first is strictly better and costs nothing. **Customer half:** new `internal/fillwatch`, per FILESYSTEM (never per app — one full disk holding ten apps would fire ten times). **It emits the PRE-EXISTING `disk_warning`/`disk_critical` pair, which was allowlisted, copy'd, in `DefaultEnabledEvents` and checkbox'd with NO PRODUCER IN ANY REPO** — a complete customer pipeline with no producer, the **sixth** *built-but-never-wired* instance here; minting a new near-duplicate type would have left it inert forever. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range — **proven live: the critical crossing fired on the FREE-BYTE term (1.7 GB) at only 91% used.** Edge-triggered on escalation, state persisted, hysteresis dead zone at 75% / 7 GiB pinned by a test; a nil usage read never warns and never clears one (§8.4). The hub's two generic `customerMessages` entries were **removed** — `FormatCustomerEmail` prefers the entry over the message, so keeping them would discard the drive label and the byte figures. **Operator half: see R-158.** **Live on 9201, all three flows:** `disk_warning` then `disk_critical` both `customer | sent` with the Hungarian rendered, exactly two events across three boots (the edge trigger held on the one between), then a silent clear that re-armed. **v0.191.1** added the once-at-startup run (Daily/Every both wait for their first tick, so a box BOOTING over the line would have stayed silent up to 24 h — the R-100 shape); **v0.191.2** added a per-run positive observable, earned when a quiet run during this session's own validation proved unreadable as evidence. Follow-ups: **R-177** (no run-now path) | — | | **R-168** | ~~CI: no runner exists, and with trunk-based pushes CI can DETECT but not BLOCK~~ | **SHIPPED — and the alarm is DEMONSTRATED** (2026-08-02) | — | **Runner live**: `homelab-manifests/gitea-system/act-runner.yaml`, an unprivileged host-mode `act_runner` in `gitea-system`, one owner-scoped registration serving all four repos (measured: tasks 7-10 all claimed by `felhom-gates-runner`). `.gitea/workflows/gates.yml` in each repo runs that repo's entry point with `--fast` and nothing else; no `uses:` step anywhere. **Six probes, all answered, none STOPped** — `audits/SPIKE-ci-runner-2026-08-02.md`. The two that changed the design: **P2** (stock image has git but NO python3 → custom image `felhom-act-runner:0.1.0`, base pinned, python3 and nothing else) and **P6** (a runner that loses `/data/.runner` re-registers and leaves a dead record behind → the PVC is load-bearing, measured both ways). **P5 is the one that mattered**: a failed run produced NO mail, NO notification row and NO log line from Gitea, so the run now sends its own alarm via Resend and prints the provider's accepted id. **Proven end to end, not asserted**: a deliberately broken commit pushed with `--no-verify` → run #6 `failure` → `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. Posture shown from the live pod spec: `privileged: false`, all caps dropped, no docker socket, no hostPath, `automountServiceAccountToken: false`, sized at half Gitea's limits so it cannot crowd out the service holding every repository on the same node. **The standing limit stays true and is written into the manifest and every workflow: it DETECTS, it does not BLOCK** — making it block is → R-169 | — | | **R-169** | **CI can only report, because there is no gate in the road.** Every felhom repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand at. R-168's runner therefore notices a broken push *after* it has landed | **WAITING-ON-OPERATOR** (a working-style decision, not a defect) | an operator ruling | Making CI *blocking* requires two things this task deliberately did NOT do, because both change how the operator works and that is not a task's call: **(a)** branch protection on `main`, and **(b)** a pull-request workflow instead of direct-to-`main` pushes. The cost is real — every change would need a PR, which for a single-operator project may be worse than the disease. **The current arrangement is two nets, and it is not nothing**: `.githooks/pre-push` REFUSES locally, and R-168's runner NOTICES when that hook was skipped or was never armed in a clone, and emails. The honest gap is the window between a `--no-verify` push landing and the operator reading the alarm. Decide only if that window ever actually costs something | operator | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 65340be..5571584 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -166,7 +166,9 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the | R-170 | **The drive-backed boot gate infers a Stop from a container count** | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | `shouldRecreateOnBoot` reads `desired_state` with the same three-way table as `isBootOrphan`; absent keeps the old `hasContainers` behaviour exactly; `presentStable` untouched. Agreement pinned from both sides against one fixture table. Live: calibre-web (`running`, zero containers) recreated and immich (`stopped`) left alone in the same reboot | | R-171 | **The boot sweep started apps whose data drive was ABSENT** — a regression from v0.189.0 | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | Reasoned from the diff, CONFIRMED on hardware first (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The write hazard was blocked only by an ACCIDENTAL filesystem permission (host-root-owned mountpoint + unprivileged guest); the false dead-app alarm was real on every box. New fail-safe `bootrecon.StartGate` seam, also covering quiesce and in-flight app-data operations (§8.2). The rule already existed on the API path (`startGatedByMissingDrive`); the sweep bypassed it | | R-166 | **App state gets a desired/observed model with its own store** (operator decision D-b) | M | **SHIPPED + PROVEN-LIVE — controller v0.189.0, 2026-08-02** | Tri-state `desired_state` in `app.yaml`, written ONLY by the customer's action; `bootrecon` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN** so legacy boxes keep byte-identical behaviour; running-only backfill; `backup.AppStopGuard` covers every stop→work→start window in its own marker file. **The durable fix for R-157 mechanism B.** Both blocking facts answered at source first: the crash-safe journal pattern DID already exist (quiesce, migrate) but covered **none** of the app-data path, and the SQLite store was rejected as a home because `metrics.db` is optional by design. `07`/`02` architecture docs carry the desired/in-flight/observed split. **Left for R-170:** the drive-backed boot gate still uses the container-count inference | -| R-158 | **A local Tier-1 backup failure reaches no hub channel** — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, only the off-box leg calls it | S | READY — 2026-08-02 | Fifth *seam built but never wired*; **R-97's defect one tier over**. Only caller `cmd/controller/main.go:659`; the manager's seams are tier2/offbox/offbox-enlarge (`backup.go:33,37,58`), none for the unit capture. **Ranked below R-157 by measurement:** `/backups` DOES show `✗ Adatmentés sikertelen`, it persists across a second failure and clears on recovery, and `/backups/apps` shows the last good unit's true mtime — so it is a notification gap, not silent failure. **The specific fix: `/backups/apps` is the page you would check for one app, and the only one that never says.** Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md`, `audits/CAMPAIGN-10-closeout-2026-08-02.md` | +| R-158 | ~~A local Tier-1 backup failure reaches no hub channel~~ | S | **SHIPPED — controller v0.191.0 + hub v0.89.0, 2026-08-02** | Collapsed per the lifecycle rule. Closed by **R-167** (D-c's operator half) — no second row was filed for the same wire. New `unitNotify` seam fired per app from `captureAllRecoveryUnits` with the loop continuing, carrying the target filesystem's used/free bytes. **Routed to a new OPERATOR-ONLY type `recovery_unit_capture_failed`, NOT the `backup_failed` this row proposed** — that type is customer-enabled by default and would email the customer about a failure they cannot act on; decision D-c overrides the proposal. **Flips:** the new capability-map row *"A failed per-app Tier-1 backup reaches the OPERATOR"* → **PROVEN-LIVE** (`customer | … | skipped | operator_only` observed in the hub's `notification_log`, 9201, 2026-08-02) | +| R-167 | ~~Storage monitoring and backup alerts (decision D-c)~~ | M | **SHIPPED — controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02** | Collapsed per the lifecycle rule. **Landed BEFORE the R-165 merge rather than with it** — D-a's condition (2) requires the same step and never after, and first is strictly better: the warnings were proven on hardware while the wall is still standing. Customer half = `internal/fillwatch`, per filesystem, two threshold terms (85% / 5 GiB; critical 95% / 2 GiB), edge-triggered with persisted state and a 75% / 7 GiB hysteresis return. **It emits the pre-existing `disk_warning`/`disk_critical` pair, which had NO producer in any repo — the sixth *built-but-never-wired* instance**; the hub's two generic `customerMessages` entries were deleted so the dynamic Hungarian survives. Operator half = R-158. **Flips:** two new capability-map rows → **PROVEN-LIVE**. **Follow-ups:** R-177 (no run-now path), R-175 (§7.5's bound is one box's) | +| R-165 | Merge `mp1` into `mp0` — the dedicated backup partition stops existing (decision D-a) | M | **SPIKED 2026-08-02 — WAITING-ON-OPERATOR** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md`, M1-M5, **no layout touched**. Prerequisite R-167 is now SHIPPED. **Three findings the merge session must not re-derive:** (1) *"the layout" is not one thing* — demo-felhom `200G/50G`, demo-hp `50G/20G`, golden `16G/8G`, so any fixed pair is already wrong for one box (→ R-175); (2) **`mp1` is a BULKHEAD, not only a ceiling** — an overflow today cannot reach `/var/lib/docker`, and after the merge it can, which is the one place "the merge is cheap" stops being true; (3) the golden fails closed on the split in **four** places, not one. **M5: D-a's condition (1) is currently SATISFIED** — no external box is in the hub's host register, and both demo boxes are Tier 0 and reinstallable, so migration cost for the measurable population is zero. **Recommendation: S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), fresh-install shape.** **Two prerequisites unmeasured → R-176.** Does NOT close R-163 until it lands | | R-159 | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy** | XS | **SHIPPED** (`templates/wishlist/`, 2026-08-02) — filed for the CLASS | Image declares `VOLUME /usr/src/app/data`; template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns names only for compose-declared volumes, so `DumpAppVolumes` never sees an anonymous one. **The class is open:** any image `VOLUME` at an unmounted path is silent unbacked-up storage — **`immich-server` has one today** at `/data`, empty when measured. Proposed `REUSE.md` rule: a template mounts every path in `Config.Volumes`, or says why not | | R-160 | **gramps-web persisted three paths and wrote to none of them** | XS | **SHIPPED** (`templates/gramps-web/`, 2026-08-02) | `/app/data` appears nowhere in the image's env; the accounts DB and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths, the template three, one a phantom. **Severity above papra's:** papra loses documents the customer may hold elsewhere; gramps-web loses the artefact built inside the app, of which no other copy exists by construction | | R-161 | **The volume-persistence gate is enforced by convention, not automatically** | M | **RULED + SHIPPED at reduced scope** (operator, 2026-08-02; `app-catalog` `fd7747d`) | Enforcement is **convention**: the catalog repo has no CI at all (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none). **R-29's record: three orphaned gates, one enforced, and only the enforced one ever stopped anything — so the ruling copied the shape that works.** **Controller-side REJECTED with a measured reason:** a check at template load can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — it would pass on the very defect it exists to catch (the property is runtime-only; see `check-volume-persistence.py`'s header). **CI REJECTED for now** — neither repo has any, no users yet. **Shipped:** `scripts/catalog_gates.py`, one entry point over all three gates, non-zero on any failure, mandated in the catalog's `CLAUDE.md` as `site_gates.py` is. **Open residue is only the automatic half**, sufficient while one person touches templates |