admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: R-167 + R-158 CLOSED, R-165 SPIKED, R-174..R-177 filed
gates / gates (push) Successful in 8s
Details

Browse Source 
R-167/R-158 shipped and proven live (controller v0.191.x, hub v0.89.0):
two new capability-map rows PROVEN-LIVE with live citations, and
07-backup-architecture.md §7.5's closing claim "nothing warns when an app
crosses the line" is now false and rewritten (S-1: an architectural
contract changed in the same session). §7.5 also gains the caveat that its
size bound is ONE BOX'S, not the fleet's.

Part 3 SPIKE (audits/SPIKE-r165-mp1-merge-2026-08-02.md): M1-M5 measured,
NO layout touched. Three findings the merge session must not re-derive:
"the layout" is not one thing (200G/50G vs 50G/20G vs 16G/8G); mp1 is a
BULKHEAD and not only a ceiling, so after the merge an overflow reaches
/var/lib/docker; the golden fails closed on the split in four places.
D-a's condition (1) is currently SATISFIED — no external box is in the
hub's register, and both demo boxes are Tier 0 and reinstallable.
Recommendation given, choice NOT made — it ends at the operator's ruling.

CONTEXT.md S-11 (D-c's routing, and why R-158's own backup_failed proposal
was overruled) and S-12 (the monitoring landed BEFORE the merge).
STATUS.md gains the plain-language section and the merge decision, with two
older entries trimmed so the page did not grow.

New rows R-174 (closed same session), R-175, R-176, R-177; each ID grepped
free before minting.
This commit is contained in:
authentik Default Admin
2026-08-02 23:56:16 +02:00
parent 6d359a5360
commit 41dbecb264
8 changed files with 478 additions and 253 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+34
View File
@@ -17,6 +17,40 @@
## Standing rulings
**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
is the OPERATOR's** (they can do none of those things about it). R-158 proposed emitting the existing
`backup_failed` for the capture failure. **That was rejected and D-c wins**, because `backup_failed`
carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents` — so reusing it emails
the customer, in Hungarian, that their backup failed, about something they cannot act on. It is
exactly the mistake R-97a avoided by minting `whole_guest_backup_failed`, and the reasoning is written
into `hub/internal/api/handler.go`'s allowlist. New type: `recovery_unit_capture_failed`, in
`allowedEventTypes` **and** `notify.operatorOnlyEvents` — **the second register is what makes it
operator-only; the first does not**, and v0.78.0 claimed otherwise and shipped the defect.
**The customer half reused the pair that already existed rather than minting a seventh type.**
`disk_warning`/`disk_critical` were allowlisted, carried Hungarian copy, sat in `DefaultEnabledEvents`
and had a UI checkbox — and **nothing in any repo emitted them**. A complete customer pipeline with no
producer: the **sixth** *built-but-never-wired* instance in this project. `internal/fillwatch` is now
that producer. Their generic `customerMessages` entries were **deleted**, because
`FormatCustomerEmail` PREFERS the entry over the message and a static template would discard the drive
label and the free-space figures — the same reason `offbox_enlarge_blocked` and `disk_health_degraded`
have none. `notify.IsOperatorOnly` was added so ONE test pins both registers; checked separately, an
allowlisted-but-not-operator-only type is invisible.
**S-12 — the monitoring landed BEFORE the merge, not with it (2026-08-02).**
D-a's condition (2) says R-167 ships in the same step as the `mp1`→`mp0` merge and never after,
because the merge removes a wall that currently fails safely. **This session landed it FIRST**, which
is strictly better and costs nothing: the warnings went in and were proven on real hardware while the
wall is still standing, so the merge session inherits a proven signal instead of an untested one.
**No disk layout was touched.** R-165's measurement is `audits/SPIKE-r165-mp1-merge-2026-08-02.md`,
which STOPS at a question for the operator (which merge shape; what replaces the bulkhead). Its two
load-bearing findings for anyone picking that up: **"the layout" is not one thing** (demo-felhom
`200G/50G`, demo-hp `50G/20G`, golden `16G/8G` — so §7.5's bound is one box's, → R-175), and **`mp1`
is also a BULKHEAD**, not only a ceiling — today an overflow cannot reach `/var/lib/docker`, and after
the merge it can.
**S-8 — CI detects; it does not block, and that is structural (2026-08-02, R-168).**
A Gitea Actions runner in `gitea-system` re-runs every repo's gate entry point on every push,
independent of who pushed and of what they typed. It **cannot refuse a push**: every felhom repo
REPORT.md
+122 -230
View File
@@ -1,250 +1,142 @@
# REPORT — CI: a Gitea Actions runner, and a red run that reaches a person (R-168, 2026-08-02)
# REPORT — hub v0.89.0: the two halves of decision D-c, plus the R-165 merge spike (2026-08-02)
**Overwritten** per the standing rule. The prior contents (session 1's gate enforcement, same day)
have their durable record in `scripts/CHANGELOG.md`, `hub/CHANGELOG.md` and `CONTEXT.md` S-6/S-7.
**Overwritten** per the standing rule. The prior contents (R-168, the CI runner, same day) have their
durable record in `scripts/CHANGELOG.md` and `CONTEXT.md` S-8/S-9/S-10.
**Outcome: complete, and the half that matters is demonstrated.** All six probes answered; none
STOPped. The acceptance test was never a green pipeline — it was a **red run with a message in
hand**, and that is `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`.
**Companion report:** `felhom-controller/REPORT.md` holds the controller side (v0.191.0/.1/.2), the
full red-proof table, the Hungarian copy, and the live evidence for all three flows. This file covers
the hub change, the documentation coupling, and **Part 3's spike**.
---
## 1. Confirmed baselines — all matched §1 on arrival
## 1. Baseline drift — recorded, because the task's §1 was wrong
| Repo | `main` on arrival | matched? | version |
The task targeted hub **v0.87.0 → v0.88.0**. On arrival `main` was at `8ef92a3f` with hub **v0.88.0
already shipped** (R-172, the WAL fix), not `d5774d318941`/v0.87.0. Target corrected to **v0.89.0**.
Highest register ID in use was **R-173**, not R-171.
## 2. Hub change (v0.89.0)
**One new event type, not two.** The task called for a new customer-facing type *and* a new
operator-only one. Reconnaissance found `disk_warning`/`disk_critical` already allowlisted here, with
Hungarian `customerMessages`, in the controller's `DefaultEnabledEvents` and behind a UI checkbox —
**and with no producer in any repo.** The operator chose to wire that inert pair rather than mint a
near-duplicate, so only the operator type is new.
| Change | File | Why |
|---|---|---|
| `+ "recovery_unit_capture_failed"` | `internal/api/handler.go` (`allowedEventTypes`) | without it the controller's POST 400s and the event vanishes |
| `+ "recovery_unit_capture_failed"` | `internal/notify/dispatcher.go` (`operatorOnlyEvents`) | **this** is what makes it operator-only; the allowlist does not, and v0.78.0 claimed otherwise and shipped the defect |
| `- customerMessages["disk_warning"]`, `- ["disk_critical"]` | `internal/notify/templates.go` | `FormatCustomerEmail` PREFERS the entry over the message, so a static template would discard the drive label and the free-space figures the controller now sends. Same reason `offbox_enlarge_blocked` and `disk_health_degraded` have none |
| `+ func IsOperatorOnly` | `internal/notify/dispatcher.go` | lets the `api` package pin BOTH registers in ONE test; checked separately, allowlisted-but-not-operator-only is invisible. Read-only — the register stays unexported so nothing can widen it at runtime |
| `REUSE.md` §5 "new event type" rewritten | `REUSE.md` | it told readers to always add a `customerMessages` entry, which is **wrong** for operator-only types and **harmful** for dynamic-message ones |
**Tests 574 → 579**, full suite green (`go build ./... && go vet ./... && go test ./...`), all five
`repo_gates.py` gates OK.
**Red-proof (Scenario G), demonstrated not argued:** removing `recovery_unit_capture_failed` from
`operatorOnlyEvents` fails two tests, one reading *"a customer was emailed the OPERATOR-ONLY
recovery_unit_capture_failed (customer@example.com)"*. The dispatch test runs under the **breaking**
configuration — the customer has the event enabled and an email set — because that is the only
configuration in which the missing entry is visible.
**Live (guest 9201 → hub):** both event types accepted and stored; `operator | sent`; and the positive
observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from
`notification_log`. The customer half: `customer | disk_warning | sent` and `customer | disk_critical
| sent` with the dynamic Hungarian intact.
**Deploy:** GitOps only — `manifests/hub.yaml` bumped 0.88.0 → 0.89.0 (`6d359a5`), pushed, then a
deliberate ArgoCD hard-refresh + sync. Never `kubectl set image`. App `felhom` **Synced / Healthy**,
`deploy/hub` rolled out, running `gitea.dooplex.hu/admin/felhom-hub:0.89.0`, startup log clean.
## 3. Part 3 — the R-165 spike. **M1-M5 each answered; nothing was changed.**
Full document: `documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md`. No partition was created,
resized, moved or deleted; no golden rebuilt; no guest config edited. `ep0` and Peti's box were not
contacted (D-d, `runbooks/target-selection.md`).
**M1 — what is actually there. ANSWERED, and it contradicts the architecture doc.**
| | demo-felhom | demo-hp | golden default |
|---|---|---|---|
| `homelab-manifests` | `420e8197769b` | yes | n/a |
| `felhom.eu` | `af2d10388021` | yes | hub `v0.87.0` — **no bump** |
| `felhom-controller` | `eaded79b1894` | yes | `v0.188.0` — **no bump** |
| `felhom-agent` | `2c4efed5de32` | yes | `v0.119.0` — **no bump** |
| `app-catalog-felhom.eu` | `f16f29757e95` | yes | n/a |
| `mp0` `/var/lib/docker` | **200 G** (13 G used) | **50 G** (5.4 G used) | 16 G |
| `mp1` `/mnt/sys_drive` | **50 G** (2.0 G used, 5%) | **20 G** (92 M used, 1%) | 8 G |
Live Gitea re-confirmation, **all matched**: version **1.26.2**; Actions enabled on **all five**
repos; **0 runners**; **0 workflow runs ever**; **0 branch protections**; **no `.gitea/`** in any
repo. ArgoCD `gitea` app: `path: gitea-system`, auto-sync **off**, Synced at `420e819`, Healthy.
The only new artifact is the runner image, at its own `0.1.0`.
§7.5 documents the appliance as `mp0 50G / mp1 20G` — that is demo-hp exactly and **not** demo-felhom.
Any merge plan expressed as a fixed pair is already wrong for one of the two boxes that exist. §7.5's
headline bound (*"≈ 19 GB … ≈ 10 GB"*) is derived from `mp1 = 20 G` and is therefore one box's, not
the fleet's → **R-175**, filed and §7.5 annotated in this session.
## 2. All six probe results
**M2 — what lives on `mp1`. ANSWERED, and it is not only backups.** Four things would move:
Tier-1 units of driveless apps (269 M, ~30 apps on demo-felhom), **Tier-2 mirrors (1.7 G — i.e. the
MAJORITY is Tier 2, not Tier 1)**, the `userdata/import` drop zone which lives on the system drive by
**contract** (R-75), and the system-data userdata namespace. Observed fill is 5% / 1%: the constraint
is a **ceiling** problem, not a current-fill one.
Full method and measurement: `documentation/audits/SPIKE-ci-runner-2026-08-02.md`.
**M3 — which merge shapes exist. ANSWERED for three shapes, with ONE item explicitly unmeasured.**
The golden **fails closed on the split in four places**, not one (`build-golden.sh:126,130`
separate-mount asserts + `:315,319` vzdump-exclusion guards). The archive scope `rootfs+mp0+mp1` stays
complete after a merge (the data moves onto `mp0`). `mountParity` holds for new archives. **Unmeasured
and reported as such:** whether a *pre-merge* archive restore-tests into a *merged* guest — reading
`mountParity` says it should, but that is reasoning from source about an unvalidated mechanism, which
this project has got wrong four times → **R-176**.
| Probe | Ruling | Measurement |
|---|---|---|
| **P1** runner picks up a job | **PASS** | run #1 claimed by `felhom-gates-runner`, `P1-OK`, `🏁 Job succeeded` |
| **P2** stock image runs our checks | **known branch → build an image** | stock `act_runner:0.6.1` (Alpine 3.23): `git 2.52.0` PRESENT, `python3` **ABSENT**. Host mode itself fine → privileged/dind never reached for |
| **P3** source without JS actions | **PASS** | `pushed sha == checked-out sha` (`bbd62319…`), plain `git clone`/`fetch` from the in-cluster Service, no `uses:` |
| **P4** one registration, four repos | **PASS** | tasks 7,8,9,10 across four repos, all `runner_id=2 felhom-gates-runner` |
| **P5** does a failure signal anything | **NOTHING SENT → build the alarm** | 0 mail/SMTP/notification log lines, **0** notification rows, after a real `failure` run |
| **P6** persistent state needed | **PVC is load-bearing** | PVC intact → same registration; `/data/.runner` removed → **second record** appears and the first is orphaned |
**M4 — the bulkhead. ANSWERED, and it is the important one.** `mp1` is not only a ceiling: today an
overflow is refused per app with the last good unit byte-identical **and cannot reach
`/var/lib/docker`**. After the merge it can, and a full Docker data-root is a stopped box, not a slow
one. Four replacements costed — a reserved block percentage, **a refusal threshold in the capture
path**, a project quota, or deeming R-167's warnings sufficient — with the trade-off of each.
**Deliberately not chosen: this is the operator's ruling.**
**P5's honest limit, stated rather than glossed:** I did not independently prove Gitea's SMTP path
can *deliver*, so "no mail" cannot be split with certainty between *no action-failure notification in
1.26.2* and *a broken mailer*. It does not change the design — the alarm rides Resend, a different
path — and Scenario C proves that path end to end.
**M5 — existing boxes. ANSWERED for the measurable population; one part honestly UNMEASURED.** The
hub's `/hosts` register holds four hosts, **two ONLINE**, both demo boxes — and both are **Tier 0,
therefore reinstallable rather than migratable (D-d)**, so migration cost for the measurable population
is **zero**. **D-a's condition (1) — "before any external install" — is currently SATISFIED**, which
makes this the cheapest this decision will ever be. **`peti-felhom` exists as a customer with NO host
in the register**, so its layout is not knowable from the hub and the box was not contacted; whether it
needs converting or reinstalling is the operator's information. The in-place migration procedure has
**never been rehearsed**, so "is the box restorable at every point of it?" is currently unknown → also
**R-176**.
## 3. Files created / modified
**Ranked options and recommendation:** (1) **S1 — one volume with the two paths as directories — plus
B2, a refusal threshold in the capture path**, shipped as a fresh-install shape with the demo boxes
reinstalled; (2) S1 + warnings only; (3) S3, grow `mp1` and keep the split (D-a's rejected baseline,
measured for comparison); (4) S2, two mounts on one pool — **not recommended at all**, it satisfies
every assertion while delivering none of the benefit and converts a clean per-app refusal into a
shared-pool exhaustion neither `df` can see coming.
**`homelab-manifests`** — `gitea-system/act-runner.yaml` (NEW: PVC + Deployment),
`gitea-system/act-runner/Dockerfile` (NEW), `CHANGELOG.md`, `REPORT.md`.
**STOPPED at the operator's question**, per the task. The merge is next session's supervised work.
**`felhom.eu`** — `.gitea/workflows/gates.yml` (NEW),
`documentation/audits/SPIKE-ci-runner-2026-08-02.md` (NEW), `CLAUDE.md`, `CONTEXT.md` (S-8…S-11),
`scripts/CHANGELOG.md`, `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
`STATUS.md`, `REPORT.md`. *(`hub/internal/web/configs.go` was broken and restored within the session
for Scenario B; it is byte-identical to its starting state — verified by the gate that objects to it.)*
## 4. Documentation coupling
**`felhom-controller`**, **`felhom-agent`**, **`app-catalog-felhom.eu`** —
`.gitea/workflows/gates.yml` (NEW), `CHANGELOG.md`, `REPORT.md`.
## 4. Commits pushed to `main`
| Repo | Hash | What |
|---|---|---|
| `homelab-manifests` | `c51b77c` | the runner manifest + Dockerfile |
| `felhom.eu` | `bbd6231` | the temporary Part 0 probe workflow |
| | `666a34d` | real `gates.yml`, probe workflow **deleted** |
| | `3252d51` | **Scenario B** deliberate breakage (`--no-verify`) |
| | `dd13f63` | alarm step v1 — died on `curl: command not found` |
| | `f7dbc33` | alarm rewritten in python3/urllib |
| | `9530de7` | alarm User-Agent (Cloudflare 1010) — **the run that produced the accepted-id** |
| | `4cc1238` | **breakage reverted**, main green |
| `felhom-controller` | `dcc400e` | `gates.yml` (with sibling clone) |
| `felhom-agent` | `eb99144` | `gates.yml` (with sibling clone) |
| `app-catalog-felhom.eu` | `aa57588` | `gates.yml` (`--fast`, no sibling needed) |
Plus the documentation commits listed at the end of this session.
## 5. Scenario results A-G
| # | Result | Positive observable |
|---|---|---|
| **A** | **PASS** | run #2 (`666a34d`) SUCCESS. **Every member gate's own stdout in the CI log**, not just the summary: `site gates OK — BOM, emoji=0…`, `hostinstall gates: ALL PASS`, `hub confirm gate OK…`, `manifest bearer gate OK…`, `OK [hostexecutor]: 61 cited paths — exact 60, suffix 1…`, then `all felhom.eu gates OK` |
| **B** | **PASS** | run #3 (`3252d51`) **FAILURE**. The failing gate's own text: `FAIL: hub/internal/web/configs.go:34 carries const hostInstallVersion…`, `hostinstall gates: 3 FAILURE(S)`, `CONVICTED: hostinstall`, `🏁 Job failed` |
| **C** | **PASS** | run #6 FAILURE → **`RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`**. See §6 |
| **D** | **PASS** | four repos, four runs, **one runner**: tasks 7/8/9/10 all `runner_id=2 felhom-gates-runner`; conclusions `success` ×4 |
| **E** | **PASS** | live pod spec (§8) **and** the refusal in a job log: `/root/.cache/act/…/3.sh: line 2: docker: command not found` → `E-OK docker refused` |
| **F** | **PASS** | catalogue run: `image-pin gate OK — 53 templates, 0 unpinned images`; `--fast SKIPPED: image-resolvable, volume-persistence…`; resolvability/volume-persistence/canary output lines **0**, image-pull lines **0** |
| **G** | **PASS** | P6: PVC intact → runner list unchanged (`1 felhom-gates-runner`); state removed → **`1` and `2`**, i.e. a second registration. Orphan deleted, `204` |
**Scenario A note worth keeping:** the controller and agent runs prove more than "it ran" — their
reuse tallies in CI (**126 exact / 6 suffix / 1 cross-repo**; **88 / 1 / 1**) are *identical* to the
local pre-push hook's. **CI and the hook agree.** Per §12 a disagreement would have outranked this
whole task; there is none.
## 6. Scenario C evidence in full
The run was genuinely red for a genuinely broken gate, and the message is the observable — not a
tick in a UI:
```
CONVICTED: hostinstall
RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab
🏁 Job failed
```
The Resend id is the **provider's accepted-id**, returned by `api.resend.com` for a message
`monitoring@felhom.eu → admin@felhom.eu`, subject `[felhom CI] gates FAILED in admin/felhom.eu`,
carrying the repo, the commit, a link to the run, and the line that matters most: *if the local
pre-push hook was green for this commit, CI and the hook disagree, and that outranks whatever the
push was for.*
**The key never appeared in the log** — Gitea masked it (`RESEND_API_KEY: ***`) and a grep for the
key prefix across the run log returns **0 lines**.
## 7. Runner deployment
- **GitOps only.** `act-runner.yaml` committed to `homelab-manifests` `main` (`c51b77c`), pushed,
then a **deliberate ArgoCD sync** of the `gitea` Application. No `kubectl apply`, no
`kubectl set image` at any point.
- **ArgoCD:** `sync=Synced health=Healthy`, revision `c51b77c68f10…`.
- **Pod:** `act-runner-…` `1/1 Running`; PVC `act-runner-data` **Bound**, 5Gi, `longhorn`.
- **Running image:** `gitea.dooplex.hu/admin/felhom-act-runner:0.1.0`, verified by deleting the local
copy and **re-pulling from the registry** (`Python 3.12.13`, `git version 2.52.0`) rather than
trusting the push's own output.
- **Online in Gitea's runner list**, with its label:
```
id=2 name=felhom-gates-runner status=online labels=[felhom-gates]
```
Owner-scoped (`owner_id=1`, `repo_id=0`) — which is why one registration serves four repos.
## 8. Security posture — shown, not asserted
From the **live** pod spec:
```
privileged : False
allowPrivilegeEscalation : False
capabilities.drop : ['ALL']
automountServiceAccountToken : False
hostNetwork/hostPID/hostIPC : None None None
nodeSelector : None
volumes : [('data', ['persistentVolumeClaim'])]
hostPath volumes : NONE
docker socket mounts : NONE
serviceaccount token mounts : NONE
resources : requests 50m/128Mi, limits 500m/512Mi
```
And the negative proven rather than inferred — from a job log:
```
/root/.cache/act/187d434f3fdbb3b4/act/workflow/3.sh: line 2: docker: command not found
E-OK docker refused (output above is the refusal)
```
Resources are **half of Gitea's limits** (Gitea: 1 CPU / 1Gi) and a fifth of its requests,
deliberately: Gitea shares this node and holds every repository. The heaviest entry point was
measured at **11.9 MiB RSS, 0.98 s wall**.
## 9. Teardown — all three layers
1. **Workload** — nothing to remove: the runner is the deliverable and stays. No scratch Deployment
or PVC was created beyond it.
2. **Host** — no scratch Longhorn volume was created beyond the runner's own 5Gi PVC, which is
intended and in git. Local Docker images (`gitea/act_runner:0.6.1` base and the built
`felhom-act-runner:0.1.0`) remain in DooPlex's local Docker store; they are ordinary build
artifacts and **no global prune was run** (forbidden on this host).
3. **The Gitea-side registration records — the layer this project has missed four times.**
- The **P6b orphan** (`id=1`) was deleted **immediately when it was created**, not left for the
end: `DELETE /api/v1/admin/actions/runners/1 → 204`.
- **Final runner list: exactly one, the intended live runner** —
`id=2 felhom-gates-runner status=online labels=[felhom-gates]`, and the live pod's own
`/data/.runner` carries `id=2`, so the record and the process agree.
**A trap worth recording, because it briefly looked like a failed teardown:** a raw
`select id,name from action_runner` returns **two** rows. Gitea **soft-deletes** runners — the
deleted row is tombstoned (`deleted` set), not removed, and the admin API correctly reports one.
The query without the tombstone filter was the misleading instrument, not the teardown. Same
class as the `| tail -5` census bug and the `go test -run` filter: **a query that does not model
the deletion semantics is not a check.**
- The **temporary probe workflow** `.gitea/workflows/probe.yml` was deleted from `main` in
`666a34d`, the same commit that added the real one.
- The **admin API access token** minted for this session (used to read run conclusions and to
delete the orphan runner) is **revoked**. The clean route — `DELETE /api/v1/users/admin/tokens/
{id}` — needs HTTP **Basic** auth with the operator's Gitea password, not the token itself;
tried `GET /api/v1/users/admin/tokens` with the `PASSWORD` entry from `~/.config/credentials`
and got **401**, so that is not the Gitea admin password. Revoked instead by deleting the single
`access_token` row I created, matched on its `cc-ci-session-*` name: **1 row before, 0 after**,
6 unrelated tokens untouched. The scratch file holding it was removed. It appears in no
committed file, log or report.
## 10. `OPEN-ITEMS.md` rows opened / closed / re-ranked
- **R-168 → SHIPPED**, with the evidence in the row: the runner, the one-registration-four-repos
measurement, the two probes that changed the design (P2, P6), and the demonstrated alarm with its
accepted-id.
- **R-29 → CLOSED**, both halves — and **closed on the demonstrated alarm, not on a green run**, as
the task required. The class it opened (*a gate that exists, is red, and is invoked by nothing*) is
answered at both ends.
- **R-161 → updated**: its automatic half now exists for the static gate. Its *original* scope, the
runtime volume-persistence gate, is deliberately still **not** automatic and should stay that way.
- **R-169 → NEW.** `grep -oE "R-[0-9]+" | sort -u -t- -k2 -n | tail` established **R-168 as the
highest in use**, so R-169 was free. *CI can only report, because there is no gate in the road* —
making it blocking needs branch protection plus a PR workflow, both of which change how the
operator works. **WAITING-ON-OPERATOR**, owner: operator. The row states the cost honestly (for a
single-operator project a PR workflow may be worse than the disease) and names the real gap: the
window between a `--no-verify` push landing and the operator reading the alarm.
- `ROADMAP.md`: R-168 collapsed to its one-liner, R-29 closed, R-169 added.
- **`architecture/00-capability-map.md` — N/A, stated rather than silently skipped.** No
customer-visible or platform capability changed; this is developer tooling.
## 11. Was `--no-verify` used?
**Yes — three times, all on `felhom.eu`, all deliberate and all disclosed.**
| Commit | Why |
| File | Change |
|---|---|
| `3252d51` | **Scenario B itself.** The commit had to carry a broken gate, and pushing it with `--no-verify` *is* the bypass CI exists to catch. The local hook would have refused it — that is the point. |
| `dd13f63` | The tree was still deliberately broken; pushing the alarm step required bypassing the hook that was correctly objecting. |
| `f7dbc33`, `9530de7` | Same, iterating on the alarm while the tree was still red on purpose. |
| `documentation/backlog/OPEN-ITEMS.md` | **R-158** closed (by R-167 — *no second row for the same wire*); **R-167** closed; **R-165** updated with M1-M5 + the operator question, stays open; **4 new rows** R-174/175/176/177 |
| `documentation/backlog/ROADMAP.md` | R-158 collapsed to a shipped one-liner; R-167 added as shipped; R-165 added as spiked/waiting-on-operator |
| `documentation/architecture/00-capability-map.md` | **two new rows**, both **PROVEN-LIVE** with live citations |
| `documentation/architecture/07-backup-architecture.md` §7.5 | **S-1: the contract changed in the same session.** The section's closing claim *"nothing warns when an app crosses the line"* is now false; the alerting is written in, and the one-box-vs-fleet caveat added |
| `CONTEXT.md` | **S-11** (D-c's routing, and why R-158's own `backup_failed` proposal was overruled) and **S-12** (the monitoring landed *before* the merge, not with it) |
| `STATUS.md` | new plain-language section; the merge decision added to *Waiting on you*; **two older entries trimmed** so the page did not grow — one screen, per its own rule |
| `REUSE.md` | the "new event type" extension point rewritten (see §2) |
Every other push in this session ran its pre-push hook and passed. The breakage was reverted in
`4cc1238`, and the hook passed on that push.
## 5. Register IDs
## 12. Observations — noticed, recorded, NOT acted on
**Opened:** R-174, R-175, R-176, R-177. Each established free by
`grep -ro "R-17n\b" documentation/ *.md` → **0 hits**, run before minting.
**Closed:** R-158, R-167, R-174. **Updated, still open:** R-165, R-163 (unchanged — it stays the
record of the constraint until the merge lands).
1. **CI and the pre-push hook never disagreed.** §12 said a disagreement would outrank the CI work;
there was none. The controller's and agent's reuse tallies in CI match the local run exactly. The
one place they *would* have disagreed was designed out rather than discovered: without a sibling
`felhom.eu` clone, both entry points fail closed with `gate is MISSING`.
2. **A near-miss that outranks a probe result.** My first census query reported Actions enabled on
five *unrelated* repos — a baseline drift big enough to change the task. It was a `| tail -5`
inside my own helper truncating the output. The measurement was never wrong; the instrument was.
This is the same class as the `go test -run` filter from session 1, and it is now a `CLAUDE.md`
rule: **an instrument that can silently drop results is not a measurement.**
3. **Two failures on the way to the alarm, both worth their write-up** — `curl: command not found`
(the image is minimal on purpose; fixed with `urllib`, not a bigger image) and **Cloudflare 403
error 1010** on `api.resend.com`, which blocks the default `Python-urllib` User-Agent. The second
matters because **it looks exactly like an auth failure**: the next person to see a 403 from
Resend should check the User-Agent before rotating a key. Both are in the spike doc and in
`CONTEXT.md` S-9.
4. **A full clone is slow enough to notice** — the P3 probe's `git clone` of `felhom.eu` took ~48 s.
The shipped workflows use `git fetch --depth 1 <sha>` instead, which is both faster and pinned to
the exact pushed commit. Not a finding, but the reason the shipped form differs from the probe's.
5. **Not acted on, out of scope:** Gitea's own mailer is configured and enabled but was never proven
to deliver (§2). If it *is* broken, nothing else in Gitea that relies on email works either —
worth a five-minute test email from the admin UI at some point. It does not affect the CI alarm,
which does not use it.
6. **Not acted on, out of scope:** `homelab-manifests` has no gate entry point and no pre-push hook,
unlike the four product repos, and now no CI workflow either. It is the repo that deploys the
cluster, so that asymmetry may deserve a look — but adding one was not in this task, and the repo
has no gates to run yet.
## 6. CI — run ids and conclusions
| Repo | Final commit | Run | Conclusion |
|---|---|---|---|
| `felhom-controller` | `9a3c485` | see below | see below |
| `felhom.eu` | (docs commit) | see below | see below |
*(Filled in from `…/actions/tasks?limit=3` after the final push — see the closing section of the
session output; a green CI that was not looked at is an assumption, not an observation.)*
## 7. `--no-verify`
**Not used anywhere.** Every push in this session ran `.githooks/pre-push` (`repo_gates.py --fast` /
`controller_gates.py --fast`) and passed.
STATUS.md
+40 -19
View File
@@ -47,11 +47,30 @@ change, not assumed. The one thing it still cannot do is *stop* the change: ever
straight to the main copy with no review step, so there is no point in the road for it to stand at.
It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)*
**A filling disk now warns the customer before anything breaks, and a failed backup now reaches you.**
Until today the first sign that a disk was filling up was a backup that did not happen — nothing said
anything beforehand. Two things changed. The customer is now warned while there is still room to act,
naming the drive and how much space is left, in plain Hungarian that says what to do about it. And
when one app's backup fails for any reason, **you** are told which app and why, with the disk figures
attached — the page that answers "is this app backed up?" was, until now, the one page that never
said. The customer is deliberately *not* told about that second one: they can free up space, but they
can do nothing about a backup that failed, so telling them would only alarm them.
Both were proven on the demo machine by actually filling a disk. One detail is worth knowing because
it is why there are two rules and not one: the serious warning fired when free space dropped below a
fixed amount while the disk was only 91% full — a percentage on its own would have missed it.
**These went in *before* the partition change deliberately.** The partition being removed is also a
barrier against a runaway backup filling the space the machine needs to run; putting the warnings in
first means that when it comes down, the thing watching is already working and already tested.
*(R-167, R-158)*
## What we're working on
- **Now:** the last app whose data was never saved; today's decisions written down.
- **Next:** merging the small backup partition into the large one, with the drive-filling warning and
the backup-failure alert in the same step.
- **Next:** merging the small backup partition into the large one — **the warnings for it are already
done and working**, so this step is now only the partition change. It needs one decision from you
first (below).
- **After:** rebuilding how the machine records whether an app is meant to be running.
## Waiting on you
@@ -63,6 +82,13 @@ It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)*
settle both. *(R-110, R-115)*
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
session log; nothing suggests anyone else saw it. *(R-132)*
- **The partition merge: one decision, now measured.** Removing the backup partition also removes a
barrier — today a runaway backup is refused on its own and cannot touch the space the machine needs
to run; afterwards it can, and a machine out of that space is stopped, not slow. So: do we add a
hard stop that refuses a backup before it eats the last of the room, or do we rely on the new
warnings? The recommendation is the hard stop, because it keeps exactly what the barrier gave us.
**Second question, which only you can answer:** does the tester's box need converting in place, or
can it be reinstalled? It does not report to the hub, so nothing here can tell. *(R-165, R-176)*
## Changed since last update
@@ -75,25 +101,20 @@ It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)*
live. **Also found while checking it: the hub's own database is not in any automatic backup** — it
holds every machine's emergency password and the escrow records. Filed, not yet fixed.
- **2026-08-02** — Boot recovery finished. Both halves of the power-cut problem are closed: the
machine records what the customer asked for, and it now waits for the system to finish starting
before deciding what is missing. Six hard resets in a row, everything back every time. A hole the
previous day's change had opened — the machine would have tried to start an app whose external
drive was missing — was found by reading the code, reproduced on the demo box **before** anything
was changed, and fixed the same day. No data ever reached the wrong disk, but only because of a
filesystem accident nothing was relying on deliberately, which is why it was fixed rather than
noted.
- **2026-08-02** — Boot recovery finished: the machine records what the customer asked for, and waits
for the system to finish starting before deciding what is missing. Six hard resets, everything back
every time. A hole the previous day's change had opened — starting an app whose external drive was
missing — was found by reading the code, reproduced on the demo box first, and fixed the same day.
**A second instance of the same hole was found and fixed today**, on the path that restarts an app
after an interrupted backup.
- **2026-08-02** — Fixed and then doubled: thirteen mechanical checks had built up across the four
repositories and nothing ran most of them; two were failing quietly, one since 14 July. Both fixed.
Every repository now has one command that runs all its checks, it runs automatically before every
push, and the build server re-runs it after every push and **emails you when it fails** — proven
with a real broken change. One of the two failures was the setup page claiming the installer was
version 1.19.0 while machines got 1.22.0; that number is now **gone** rather than corrected,
because the page cannot know it — the installer is fetched fresh each run, so any number there is a
guess. A check refuses to let one be added back.
- **2026-08-02** — Thirteen mechanical checks had built up across the four repositories and nothing
ran most of them; two were failing quietly, one since 14 July. Both fixed, and the arrangement that
replaced them is described above.
- **2026-08-02** — Decided: the 20 GB backup partition goes away and shares space with app data. That
changes the disk layout, so it happens before any machine is installed outside the house.
changes the disk layout, so it happens before any machine is installed outside the house. **Measured
since:** no machine outside the house is registered yet, so this is as cheap now as it will ever be;
and the two demo machines can simply be reinstalled rather than converted.
- **2026-08-02** — Decided: only this machine and the tester's box are protected; every other box,
demo boxes included, may be broken or reinstalled freely. Two of the three apps that never saved
their data are fixed; this page created.
documentation/architecture/00-capability-map.md
+2
View File
@@ -84,6 +84,8 @@
| Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) |
| An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 |
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
| **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait |
| **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it |
| Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise |
documentation/architecture/07-backup-architecture.md
+22
View File
@@ -529,6 +529,28 @@ portable secrets in the unit. **That independence is bounded by app size**, and
> Past that the unit cannot be captured, and the app falls back to Lane 2's operator-driven
> whole-guest route.
**AS OF 2026-08-02 SOMETHING NOW WARNS, AND THE ALERTING IS PART OF THIS CONTRACT (R-167 / R-158,
decision D-c; controller v0.191.x + hub v0.89.0).** The last sentence of this section used to end
"nothing warns when an app crosses the line". Two signals now exist and both are PROVEN-LIVE:
- **To the CUSTOMER, before anything fails** — `internal/fillwatch` warns per FILESYSTEM (never per
app: one full disk holding ten apps would fire ten times) on **whichever trips first, used ≥ 85% or
free < 5 GiB**, critical at 95% / 2 GiB, clearing at 75% / 7 GiB. **Two terms, because a percentage
alone lies at both ends of the range this section itself documents:** 85% of a 20 G `mp1` leaves
3 G — less than one DB-backed app's unit — while 85% of a 4 TB drive leaves 600 G. It watches the
app-data volume, the system-data volume **and** every registered drive, which the previous
`health_degraded` signal did not. Edge-triggered against persisted state; the hub owns cooldown.
- **To the OPERATOR, when a capture actually fails** — `recovery_unit_capture_failed`, per app, with
the target filesystem's used/free bytes at the moment of failure, so the *why* needs no login. It is
**operator-tier** (`notify.operatorOnlyEvents`) and deliberately not `backup_failed`: a customer can
take no action on a capture failure.
**A caveat this section must carry, because the bound below depends on it.** The `mp0 50G / mp1 20G`
table above is **demo-hp's** shape, not the fleet's — demo-felhom ships `mp0 200G / mp1 50G`, where
the same bound is ≈ 49 GB / ≈ 24 GB, and the golden's own defaults are `16 G / 8 G` before provision
grows them. **The bound below is a FUNCTION of `mp1`, not a constant.** Measured 2026-08-02,
`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1; correcting the numbers throughout is **R-175**.
Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling**
(**R-163**) — this section states the constraint, not a number. And **the same-device placement is
intended, not a defect**: a driveless app's unit sits on the same SSD as its volumes, but Tier-2's
documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md
+248
View File
@@ -0,0 +1,248 @@
# SPIKE — measuring the `mp1` → `mp0` merge (R-165, decision D-a)
**Date:** 2026-08-02 · **Author:** Claude Code · **Status:** MEASUREMENT ONLY — **no layout changed**
> **Decision D-a is already taken** (`CONTEXT.md` S-5): the dedicated backup partition is merged away
> rather than resized. This document measures **how**, never **whether**. It ends in a question for the
> operator and a **STOP** — the merge itself is next session's supervised work.
## 0. What was and was not touched
Nothing was created, resized, moved or deleted. No golden was rebuilt, no guest config edited, no
partition altered. Every figure below is a read of live state or of committed source. `ep0` and Peti's
box were not contacted at all (`runbooks/target-selection.md`, decision D-d).
The monitoring D-a requires (**R-167**) shipped **before** this measurement, in controller v0.191.x and
hub v0.89.0, and was proven on guest 9201. That ordering is the point: D-a's condition (2) says the
monitoring lands with the merge and never after, and landing it first is strictly better.
---
## M1 — what is actually there
**`pct config 9201`, both demo hosts, read 2026-08-02.**
| | demo-felhom (N100) | demo-hp (t740) | golden default |
|---|---|---|---|
| `rootfs` | 32 G | 32 G | `OS_SIZE_GB=32` |
| `mp0` `/var/lib/docker` | **200 G**, `backup=1` | **50 G**, `backup=1` | `GOLDEN_DOCKER_GB=16` |
| `mp1` `/mnt/sys_drive` | **50 G**, `backup=1` | **20 G**, `backup=1` | `GOLDEN_SYSDATA_GB=8` |
| `mp8` `/mnt/felhom-drives` | bind | bind | — |
| `mp9` bootstrap | bind, `ro=1` | bind, `ro=1` | — |
**Occupancy (`df`, in-guest):**
| filesystem | demo-felhom | demo-hp |
|---|---|---|
| `/` | 945 M / 32 G (4%) | 942 M / 32 G (4%) |
| `/var/lib/docker` (`mp0`) | 13 G / 197 G (7%) | 5.4 G / 50 G (12%) |
| `/mnt/sys_drive` (`mp1`) | **2.0 G / 50 G (5%)** | **92 M / 20 G (1%)** |
| data drive | 21 G / 916 G (3%) — `hdd_1` | 6.7 G / 938 G (1%) — `nvme-1tb` |
**THE FIRST FINDING IS THAT "THE LAYOUT" IS NOT ONE THING, AND THE SPREAD IS WIDER THAN §7.5 RECORDS.**
`architecture/07-backup-architecture.md` §7.5 states the default appliance as `mp0 50G / mp1 20G` —
that is demo-hp exactly, and it is **not** demo-felhom, which ships **200 G / 50 G**, four and two and
a half times larger. Both differ again from the golden's own `16 G / 8 G`, because `provision` grows
the volumes after restore. **Any merge plan expressed as a fixed pair of numbers is already wrong for
one of the two boxes that exist.**
**The corollary matters more than the numbers.** §7.5's headline bound — *"≈ 19 GB of app data for a
file-only app, ≈ 10 GB for a DB-backed one"* — is derived from `mp1 = 20 G`. On demo-felhom, where
`mp1` is 50 G, the real bound is ~49 GB / ~24 GB. **The architecture doc states one bound as if it
were the fleet's, and it is one box's.** That is a documentation defect independent of the merge and
is filed as its own row.
---
## M2 — what lives on `mp1` (it is not only backups)
**`du`, in-guest, read 2026-08-02.**
```
demo-felhom /mnt/sys_drive/felhom-data 2.0 G
├── backups/primary 269 M Tier-1 units of DRIVELESS apps (~30 apps)
├── backups/secondary 1.7 G Tier-2 mirrors
└── userdata/import 12 K the canonical drop-zone (R-75)
demo-hp /mnt/sys_drive/felhom-data 92 M
├── backups/primary uptime-kuma, paperless
├── backups/secondary paperless-ngx
└── userdata/import paperless
```
**Four distinct things would move, not one**, confirming the task's warning that a plan accounting
only for the units is wrong:
1. **`backups/primary/<app>`** — the RETAINED Tier-1 recovery unit of every app with no data drive.
Thirty apps on demo-felhom.
2. **`backups/secondary/<app>`** — Tier-2 cross-drive mirrors, including the `_shares` pseudo-stack.
On demo-felhom this is **1.7 G of the 2.0 G — the majority is Tier 2, not Tier 1.**
3. **`userdata/import`** — the canonical, app-INDEPENDENT drop zone (`GetImportRoot`, R-75). It lives
on the system drive **by contract**, not by convenience.
4. **`userdata/`** more generally — the system-data userdata namespace for driveless apps.
**Observed occupancy is far below capacity on both boxes** (5% and 1%). The measured pressure today is
zero; the constraint R-163 records is a *ceiling* problem, not a *current fill* problem. That is worth
stating plainly because it bears on urgency, not on correctness.
---
## M3 — which merge shapes exist, and what each breaks
Three interacting assertions exist today, and a merge touches all three. **All three were read at
source; none had been measured before.**
**(a) The golden build ASSERTS the split and ABORTS if it is absent.**
`felhom-agent/configs/build-golden.sh:130`:
```
findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . || {
echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }
```
There is a sibling assertion for `mp0` at :126, and **two more** at :315-324 that abort if the vzdump
log shows `excluding volume mount point mp0` or `mp1`. So the golden build fails closed on the split
in **four** places, not one.
**(b) The whole-guest archive's scope is `rootfs + mp0 + mp1`,** and only because both carry
`backup=1`. `build-golden.sh:69` records why: *"backup=1 is MANDATORY: without it vzdump EXCLUDES the
volume (extra mountpoints default backup=0)"*.
**(c) `mount_parity` compares the ARCHIVE's `mpN` set against the RESTORED guest's**
(`felhom-agent/internal/reconcile/restoretest.go:271-283`, `mountParity` at :347). Per slot it requires
the same mount path and a restored size **not smaller** than the archive's, and it checks the reverse
direction too. A mismatch fails the restore-test outright.
| shape | (a) golden assertion | (b) archive scope | (c) `mount_parity` | verdict |
|---|---|---|---|---|
| **S1 — one volume, two directories.** `mp1` stops existing; `/mnt/sys_drive` becomes a directory on the `mp0` filesystem | **BREAKS** — :130 aborts the build. Must be deleted, and :319's vzdump guard for `mp1` with it | **HOLDS** — scope becomes `rootfs + mp0`, still complete, because the data moved onto `mp0` | **HOLDS for new archives** (no `mp1` in archive ⇒ none required in restore). **A pre-merge archive restored into a merged guest is a different question — see below** | the shape D-a describes |
| **S2 — two mounts, one backing pool.** `mp0` and `mp1` remain separate `mpN` slots on the same storage | **HOLDS** — both are still separate mounts | **HOLDS** unchanged | **HOLDS** unchanged | **does NOT remove the ceiling** — two filesystems still have two independent `df`s. This is thin-provisioning, not a merge, and it converts a clean per-app refusal into a shared-pool exhaustion that neither volume can see coming |
| **S3 — grow `mp1`, keep the split** (**the shape D-a REJECTED — measured here as the baseline**) | **HOLDS** | **HOLDS** | **HOLDS** | zero structural risk, one `--sysdata-grow` value. It is *"the same wall further away"* — D-a's own words — and does not close R-163 |
**The one genuinely unmeasured item in M3, stated as unmeasured:** whether a **pre-merge archive**
(carrying `mp1`) restores cleanly into the merged world. Reading `mountParity` says it should — the
restore recreates `mp1` from the archive, so archive and restored guest agree, and parity passes. But
**that was reasoned from source and not executed**, and this project's own record is that four
production designs specced against unvalidated mechanisms were all wrong. **It is a one-command
restore-test on a Tier-0 box and should be run before the merge, not after.**
---
## M4 — the bulkhead question (the important one)
**Today `mp1` is not only a ceiling; it is a BULKHEAD.** An app whose recovery unit outgrows the space
is refused **per app**, its last good unit is preserved **byte-identical** (R-158's measurement), and
— critically — **the overflow cannot reach `/var/lib/docker`**, because it is a different filesystem.
The container runtime keeps running.
After S1 the same overflow lands on the filesystem Docker itself runs on. A runaway recovery-unit
capture would fill `/var/lib/docker`, and a full Docker data-root is not a degraded state, it is a
stopped one.
**This is the one place where "the merge is cheap" stops being true**, and it is why the warnings
shipped first rather than alongside.
Four candidate replacements. **No choice is made here — this is the operator's ruling.**
| # | replacement | what it costs | what it leaves open |
|---|---|---|---|
| **B1 — a reserved block percentage on the merged filesystem.** `tune2fs -m` reserving N% for root; the controller runs unprivileged, so a capture cannot consume the reserve while root-owned Docker can | one `tune2fs` at build time; no code | the reserve protects *root*, not *Docker's runtime need* specifically; sizing it is a guess without a measured worst case |
| **B2 — a refusal threshold in the capture path.** `captureAllRecoveryUnits` refuses when free space would drop below a floor, per app, and emits the alert R-158 just wired | small, local, testable; **reuses the alert that now exists** and restores the per-app-refusal semantics the bulkhead gave for free | the floor is a number needing justification; it protects the capture path only — a customer filling `mp0` through app data is untouched by it |
| **B3 — a filesystem quota on the backup directory.** XFS project quota / ext4 project quota on `<mp0>/sys_drive/backups` | enforces at the filesystem, so **every** writer is bounded, not only the capture path | recreates a fixed ceiling — i.e. it is R-163 again inside one volume, and D-a's objection to *"a bigger number is the same wall further away"* applies to it word for word |
| **B4 — R-167's warnings are deemed sufficient.** No hard stop; the customer is warned at 85% / 5 GiB and critically at 95% / 2 GiB | zero — it is already shipped and proven live | **a warning is not a bulkhead.** It depends on a human acting within the window, and the failure it fails to prevent is "Docker's data-root is full", which is the worst failure on the box |
**The measured input to that ruling:** on demo-felhom `mp1` holds 2.0 G against a 50 G ceiling, and
`mp0` has 175 G free. The overflow scenario is not close today on either box. **B2 is the only option
that preserves the property the bulkhead actually provided** — a per-app refusal with the last good
unit intact — and it is the one that reuses what R-158 just built.
---
## M5 — existing boxes, and what a migration costs
**Read from the hub's own registers, 2026-08-02. No box was contacted.**
The hub's `/hosts` register holds **four** hosts, of which **two are ONLINE**:
| host | customer | agent | status |
|---|---|---|---|
| `demo-felhom-8363b5` | Demo Ügyfél | 0.119.0 | **ONLINE** |
| `demo-hp-bb76ea` | Demo HP | 0.119.0 | **ONLINE** |
| `drill-r50-0a4f9a` | drill-r50 | 0.113.0 | DOWN |
| `sess-f-2670b5` | R-120 golden 0.186.0 proof | 0.116.0 | DOWN |
**The customer register lists five customers** — `david`, `demo-felhom`, `demo-hp`, `peti-felhom`,
`sess-f`.
**THE FINDING THAT CHANGES THE COST: `peti-felhom` EXISTS AS A CUSTOMER BUT HAS NO HOST IN THE
REGISTER.** Consistent with the long-standing *"guest not on agent node"* stop in the Peti-return
runbook. So:
- **The two demo boxes are the entire measurable migrated population, and both are Tier 0 —
disposable, per decision D-d.** For them a "migration" is not required at all: they can be
**reinstalled** from a merged golden, which is cheaper and lower-risk than migrating in place, and
D-d explicitly permits it.
- **D-a's condition (1) — "it must land before any external install" — is currently SATISFIED.** No
external box appears in the hub's host register. **This is the cheapest this decision will ever be,
and the window is open now.**
**UNMEASURED, and reported as unmeasured rather than omitted:** Peti's box's actual disk layout. It
does not report to the hub, so the hub holds no `pct config` for it, and it is protected by D-d and
`target-selection.md` — so it was not contacted. **Whether a merged golden implies an in-place
migration for that box, and what that costs, is not established by this spike.** Nor is whether the
box is currently restorable at every point of such a migration. Both are inputs the operator has and
this session does not.
**Also unmeasured:** the per-box in-place migration procedure itself (move `<mp1>/felhom-data` onto
`mp0`, drop the `mp1` slot, verify) has not been executed even once on a throwaway guest. If an
in-place migration is ever needed, **that rehearsal is the first thing to do, and its "is the box
restorable at every point?" answer is currently unknown.**
---
## Ranked options, and a recommendation
**Ranked by what the measurements support, not by preference.**
1. **S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), shipped as a
FRESH-INSTALL shape, with the demo boxes REINSTALLED rather than migrated.**
It is what D-a describes; it genuinely removes the ceiling rather than moving it; the four golden
assertions and the two vzdump guards are a bounded, greppable edit; the archive scope stays
complete; and B2 restores the per-app refusal that is the bulkhead's real value, reusing the alert
R-158 just wired. The migration cost for the measurable population is **zero**, because both boxes
are Tier 0 and reinstallable.
2. **S1 + B4 (warnings only).** Cheapest, and everything it needs is already shipped and proven live.
Rejected as the *recommendation* only because the failure it declines to prevent — a full Docker
data-root — is the worst one on the box, and it depends on a human acting inside the window.
3. **S3 (grow `mp1`, keep the split).** Zero structural risk, one number. It is the measured baseline
and D-a rejected it; recorded so the decision is compared against something.
4. **S2 (two mounts, one pool).** Not recommended at all. It keeps both assertions satisfied while
delivering none of the benefit, and converts a clean per-app refusal into a shared-pool exhaustion
neither `df` can see coming — strictly worse than today.
**Two things to do BEFORE the merge session, both cheap, both currently unmeasured:**
- **Run one restore-test of a PRE-MERGE archive into a merged-layout guest on a Tier-0 box.** M3
reasons it passes; nothing has executed it. This project's own record is that reasoning from source
about an unvalidated mechanism has been wrong four times.
- **Rehearse the in-place migration once on a throwaway guest**, and record whether the guest is
restorable at every point of it. Only needed if Peti's box turns out to require migrating rather
than reinstalling — which is the operator's information, not the hub's.
---
## THE QUESTION FOR THE OPERATOR
**Which shape, and what replaces the bulkhead?**
The recommendation is **S1 + B2**, fresh-install shape, demo boxes reinstalled. The two open inputs
only the operator has are:
1. **Does Peti's box need an in-place migration, or can it be reinstalled?** The hub cannot answer
this — the host is not in its register. It decides whether the migration rehearsal is required work
or optional insurance.
2. **Is B2's per-app refusal the right replacement for the bulkhead, or is B4 (warnings only)
acceptable?** B2 costs a small amount of code and a justified floor; B4 costs nothing and is
already live, but accepts that a full Docker data-root is reachable.
**STOP.** The merge is next session's work, and it is a supervised one.
documentation/backlog/OPEN-ITEMS.md
+7 -3
View File
@@ -89,17 +89,21 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| **R-170** | ~~**The drive-backed boot gate infers a customer's Stop from a container count.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | `shouldRecreateOnBoot` now reads `desired_state` with the SAME three-way table as `isBootOrphan`: `stopped` → never; `running` → recreate whatever the container count; **absent → exactly the pre-v0.190.0 `hasContainers` behaviour**. `presentStable` untouched and still load-bearing (an absent drive is never recreated here — the very term the boot sweep was missing, R-171). Its comment argued at length FOR the container count and was rewritten; a correct implementation under a comment arguing the opposite is worse than either alone. **The agreement is pinned from BOTH sides** against one fixture table (`TestBothBootGatesAgreeOnIntent` / `TestShouldRecreateOnBoot_AgreesWithBootrecon`) because the two gates cannot be called from one package without an import cycle. **Live on 9201, both halves in one reboot:** calibre-web (drive-backed, `running`, ZERO containers) → `recreating drive-backed app calibre-web`; immich (`stopped`) → `1 drive-backed app(s) left stopped on purpose` | — |
| **R-171** | **The boot sweep started apps whose data drive was ABSENT — a regression introduced by v0.189.0, now FIXED.** Replacing `isBootOrphan`'s container-count term with recorded intent made a drive-gate-stopped app (`compose down` ⇒ zero containers, and the gate never touches `desired_state` because it is not the customer) read as a boot orphan | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | **Reasoned from the diff, then CONFIRMED on hardware before any fix was written** (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The sweep found and started calibre-web with its drive unmounted, burned both attempts and handed it to the dead-app alarm — **a false alarm about an app the drive gate is deliberately holding**. The *write* hazard did NOT materialise: compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is host-root-owned and the guest is unprivileged — **an accidental protection no code owns, no test pins, and one `chown` or one privileged guest away from gone**. Fix: new consumer-side seam `bootrecon.StartGate`, **fail-safe (cannot determine ⇒ do not start)**, wired in `main.go`; `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two cannot drift. **The rule is not new** — the API's `startGatedByMissingDrive` already refused this to the customer; the sweep bypassed it by calling `Manager.StartStack` directly. Widening the window (R-157 A) made two more holders reachable, so the same seam also refuses an app held by a **quiesce** or an **in-flight app-data operation** (§8.2), reusing `quiesce.SuppressedStacks()` and a new read-only `AppStopGuard.HeldStacks()`. Held apps report as `HeldByDrive`, never `StillDown` — that is the alarm's bucket. **ID established free:** `grep -ro "R-171\b" documentation/ *.md` → 0 hits before minting | — |
| **R-172** | ~~**A false `host_stale` alarm fires when the hub's SQLite refuses two consecutive host reports.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (hub v0.88.0, 2026-08-02) | — | **ROOT CAUSE WAS NOT TUNING — THE PRAGMAS WERE NEVER APPLIED.** `store.New` used `?_journal_mode=WAL&_busy_timeout=5000`, which is **mattn/go-sqlite3** syntax; the driver is **modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`/`_time_format`/`_time_integer_format`/`_txlock`/`_inttotime` and **ignores the rest without an error**. The hub ran in rollback-journal mode with `busy_timeout=0` for its entire life while its own source said WAL — a configuration asserting an invariant the code did not provide. Proof: a 128 MB open `/data/hub.db` with **no `-wal`/`-shm` beside it**. **Fix:** `?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate`. **`_txlock=immediate` is not optional** — `database/sql`'s `Begin()` is DEFERRED, so a read-then-write tx must upgrade its lock and a failed upgrade is `SQLITE_BUSY_SNAPSHOT`, which **`busy_timeout` does not retry**; this store has 10+ `db.Begin()` sites, all write paths. **Retry options (b) and (c) were deliberately NOT taken** — with readers no longer blocking writers a surviving `SQLITE_BUSY` would be a real signal, and a retry would hide it; revisit only on evidence. **Live:** `-wal`+`-shm` now present, **zero `SQLITE_BUSY` since rollout**, host back to `ok`, and `PRAGMA integrity_check` = `ok` with `journal_mode=wal` after three unrelated OOM restarts. **Operational consequence handled:** a WAL DB cannot be copied by taking `hub.db` alone — the break-glass retrieval in `operations/nodes.md` did exactly that and is now WAL-aware (the live `-wal` was 729 KB, i.e. a bare `cat` would have silently omitted it) | — |
| **R-174** | ~~**The app-stop guard's crash recovery started apps onto MISSING drives — a regression in v0.189.0 code.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0, 2026-08-02) | — | **Found by REVIEW on 2026-08-02, in code shipped 2026-08-01, and closed the same session — R-171 one path over.** `appStopGuard.SetStarter(stackMgr)` handed `Recover` the RAW stack manager, whose `StartStack` has no drive gate, and `Recover` runs **at startup** — exactly when an external drive may not have come back. So: a backup stops an app, the box loses power, the drive does not remount, and the app is started on a missing drive. The rule was not new — the API's own `startGatedByMissingDrive` already refused this to the customer; the guard bypassed it. **`bootDriveGate` could NOT be reused whole**, and the reason is recorded in the code: its holder #2 reads `bootAppStopGuard.HeldStacks()`, which during `Recover` is **the guard's own marker** — it would refuse every recovery it was meant to perform — and holders #1/#2 read package-level vars assigned AFTER `Recover()` runs, so a whole-gate reuse would be correct only by accident of nil-safety. Holder #3 is extracted into a shared `driveStartGate` with **two callers, one implementation**, and `TestBootDriveGateAndAppStopShareTheDrivePredicate` pins the delegation. **A REFUSAL IS NOT A FAILURE:** new `ErrStartRefused` + a `Refused` bucket — both keep the marker, only `Failed` alarms, because routing a deliberate hold into `NotifyBackupFailed` (customer-enabled by default) is the very R-171 false alarm this fixes. `main.go` guards on `Alarming()`, not `!= nil`, and the pre-existing seam test was TIGHTENED to require it. **Live on 9201, both directions:** drive held unmounted → `refusing to restart "calibre-web" … drive /mnt/felhom-drives/hdd_1 is not a live mountpoint`, marker retained byte-identical, zero containers started, `not alarming`; drive returned → `restarted calibre-web`, marker CLEARED. **ID established free:** `grep -ro "R-174\b" documentation/ *.md` → 0 hits | — |
| **R-175** | **`07-backup-architecture.md` §7.5 states ONE box's size bound as if it were the fleet's.** Its headline — *"an app can be restored by Lane 1 only while its recovery unit still fits the retained space — ≈ 19 GB of app data for a file-only app, ≈ 10 GB for a DB-backed one"* — is derived from `mp1 = 20 G`. **That is demo-hp exactly, and it is not demo-felhom**, which ships `mp0 200G / mp1 50G`, where the real bound is ≈ 49 GB / ≈ 24 GB. The golden's own defaults are a third pair (`GOLDEN_DOCKER_GB=16` / `GOLDEN_SYSDATA_GB=8`), grown at provision time | **READY (S) — NEW 2026-08-02** | — | **Measured, not inferred** (`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1: `pct config 9201` on both hosts). Independent of the merge — the sentence is wrong today and will be wrong differently after R-165. **The fix is to state the bound as a FUNCTION of `mp1`, not a constant**, and to say which box any quoted figure came from. Same class as the comment-asserting-an-invariant rule: a doc stating a fleet-wide number that only one machine satisfies reads as settled and is not. **ID established free:** `grep -ro "R-175\b" documentation/ *.md` → 0 hits | CC |
| **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `<mp1>/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **READY (S) — NEW 2026-08-02** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits | CC |
| **R-177** | **There is no operator-triggerable "run the fill check now" path.** `fill-watch` is reachable only on its daily 03:30 schedule plus the once-at-startup run added in controller v0.191.1 — so the only way to exercise it on demand is to restart the controller | **READY (S) — NEW 2026-08-02** | — | **Noticed while live-validating R-167 on 9201, not by a failure.** It cost a controller restart per observation during validation, and it costs the same on a support call: after a customer frees space, nobody can confirm the warning has cleared without restarting their controller or waiting until 03:30. **Partially mitigated already** — v0.191.2 makes every run log a positive observable (`checked N filesystem(s), M unreadable/skipped, K notification(s)`), so at least a run that DID happen is visible; the gap is triggering one. The scheduler has `GetJobs` but no run-now, so this is a general affordance, not a fill-watch one — **scope it as "run a named scheduler job now", operator-gated.** **ID established free:** `grep -ro "R-177\b" documentation/ *.md` → 0 hits | CC |
| **R-173** | **The hub's SQLite PVC is excluded from every Longhorn backup job.** `pvc/hub-data` carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly` (04:00 / Sun 05:00) are the ONLY recurring jobs and both target the `default` group — so the 128 MB `/data/hub.db` has **no volume-level backup**. That database holds `host_recovery` (every managed box's break-glass root password), `host_escrow` + `host_escrow_superseded` (escrow custody), `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg endpoints/peers — i.e. the material several documented recovery routes depend on | **READY (M) — NEW 2026-08-02** | — | **Noticed while checking the blast radius of the R-172 WAL change, not by a failure** — the WAL work needed to know who copies this file, and the answer turned out to be nobody on a schedule. **Establish before designing:** (a) whether the exclusion is deliberate (a 1 Gi RWO Longhorn volume snapshotting a 128 MB SQLite file is cheap, so the label looks like a leftover rather than a decision) and by whom; (b) whether anything else backs it up out-of-band that this census missed — the `_recovery-inventory-2026-07-28.md` records a MANUAL hot copy, which is not a backup. **When it is designed, it must be WAL-aware** (R-172): a volume snapshot of a live WAL database is crash-consistent and replays on open, which is fine, but any file-level copy must take `hub.db-wal` too or it silently loses the newest writes. **Grep establishing the ID was free:** `grep -ro "R-173\b" documentation/ *.md` → 0 hits | CC |
| **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC |
| **R-158** | ~~**A local Tier-1 app-data backup failure reaches no hub channel.**~~ | **CLOSED BY R-167 — SHIPPED + PROVEN-LIVE** (controller v0.191.0 + hub v0.89.0, 2026-08-02) | — | **Closed by the wire it named; no second row was filed for it** (R-167 subsumes and widens it). New `unitNotify` seam + `SetUnitNotify` beside the manager's existing three, called from `captureAllRecoveryUnits` **per app with the loop continuing**, carrying the target filesystem's used/free bytes at the moment of failure — the cause is usually a full filesystem and those numbers answer *why* without an operator logging in. **ROUTED TO THE OPERATOR, NOT `backup_failed`, AND THAT OVERRIDES THIS ROW'S OWN PROPOSAL.** The proposal above said *"emitting the existing `backup_failed`"*; that type carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents`, so it would email the customer in Hungarian about a failure they cannot act on — precisely the mistake R-97a avoided by minting `whole_guest_backup_failed`. **Decision D-c routes it to the operator and D-c wins.** New `recovery_unit_capture_failed` in `allowedEventTypes` **and** `notify.operatorOnlyEvents`; `notify.IsOperatorOnly` added so ONE test pins both registers (allowlisted-but-not-operator-only is invisible when they are checked separately — the v0.78.0 defect). **Red-proof:** removing the register entry shows the customer being emailed. **Live on 9201:** two events accepted and stored, `operator | sent`, and the positive observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from the hub's `notification_log` | — |
| **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `<project>_<name>` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC |
| **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |
| **R-161** | **The volume-persistence gate is enforced by CONVENTION, not automatically.** The catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none exists). | **REDUCED SCOPE — open** (operator ruling 2026-08-02) | a second person touching templates | **RULED. Both obvious enforcement points were rejected for measured reasons.** *Controller-side at template load:* rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — **it would pass on the exact defect it exists to catch**; the property is decidable only at runtime. *CI:* rejected for now — neither repo has any, and there are no users yet. **SHIPPED instead** (`app-catalog-felhom.eu` `fd7747d`): `scripts/catalog_gates.py`, ONE entry point running all three gates, non-zero exit on any failure, **mandated in the catalog's `CLAUDE.md`** the way `site_gates.py` is. Rationale for the record: of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md — `site_gates.py` is run, R-29's three orphans are named nowhere and have stopped nothing. **What remains open is only the automatic half:** this is convention, run by a person, and that is sufficient while one person touches templates. Revisit when a second does **UPDATE 2026-08-02:** `catalog_gates.py` gained `--fast` (gate 1 only — the network and runtime gates are deliberately NOT in a hook: a push that pulls images and starts containers gets bypassed within a week and the bypass becomes the habit) and `.githooks/pre-push` now runs it. **The automatic half now has a designated successor row: R-168** (Gitea Actions runner). This row stays open at its reduced scope — the runtime gate remains a deliberate periodic run **UPDATE 2026-08-02 (second):** the automatic half now EXISTS — R-168's runner executes `catalog_gates.py --fast` on every push to this repo (measured: run #1, `image-pin gate OK — 53 templates`, with the two runtime gates announced as skipped and their own output absent from the log). This row's *original* scope — the RUNTIME volume-persistence gate — is deliberately still NOT automatic and should stay that way: CI that pulls 53 images on every push gets disabled. It remains a periodic run | operator |
| **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC |
| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio** | — (the sizing question is answered; the work is **R-165**) | **RE-FRAMED, NOT CLOSED (operator decision D-a, 2026-08-02 — `CONTEXT.md` S-5).** The row asked *what ratio should `mp1` be?* and that question is **withdrawn rather than answered**: `mp1` is merged into `mp0` so local recovery units share the app-data area and the ceiling stops existing — a bigger number is the same wall further away. **This row stays open as the record of the constraint** (what `mp1` is for, what it gates, and the measured 2× DB-app unit size) **until the merge lands**, because until then every consequence below is still live on every box. **The work is R-165; the warning that must ship with it is R-167.** Original finding, unchanged, follows. **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 (widened to R-167) | CC |
| **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
| **R-165** | **Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.** Operator decision **D-a**, 2026-08-02 (`CONTEXT.md` S-5). Local recovery units share the app-data area instead of holding their own fixed ceiling, so the wall R-163 describes is removed rather than moved further away. Guest 9201 on demo-hp ships `mp0 50G / mp1 20G` today | **READY (M) — NEW 2026-08-02** | — | **Two conditions travel WITH the decision and are not optional.** **(1) Before any external install.** It changes the **disk layout**, so it is a fresh-install shape while there are no external boxes and a per-box migration after — and the decision's cheapness is entirely a function of that ordering. **(2) It removes a wall that currently fails safely**, so **R-167** (D-c: fill warning + failure alert) lands in the same step, never after: today an app that outgrows `mp1` is refused per app with the last good unit preserved byte-identical (R-158's measurement), and after the merge the same overflow consumes the space the app itself is using. Touches the installer/agent guest shape (`--sysdata-grow` defaults to **0** and is not derived from the physical drive, `felhom-agent/cmd/felhom-agent/main.go:178`) and the golden. **Does NOT close R-163** — that row is the record of the constraint and stays open until this lands | CC |
| **R-165** | **Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.** Operator decision **D-a**, 2026-08-02 (`CONTEXT.md` S-5). Local recovery units share the app-data area instead of holding their own fixed ceiling, so the wall R-163 describes is removed rather than moved further away. Guest 9201 on demo-hp ships `mp0 50G / mp1 20G` today | **READY (M) — NEW 2026-08-02** | — | **Two conditions travel WITH the decision and are not optional.** **(1) Before any external install.** It changes the **disk layout**, so it is a fresh-install shape while there are no external boxes and a per-box migration after — and the decision's cheapness is entirely a function of that ordering. **(2) It removes a wall that currently fails safely**, so **R-167** (D-c: fill warning + failure alert) lands in the same step, never after: today an app that outgrows `mp1` is refused per app with the last good unit preserved byte-identical (R-158's measurement), and after the merge the same overflow consumes the space the app itself is using. Touches the installer/agent guest shape (`--sysdata-grow` defaults to **0** and is not derived from the physical drive, `felhom-agent/cmd/felhom-agent/main.go:178`) and the golden. **Does NOT close R-163** — that row is the record of the constraint and stays open until this lands **MEASURED 2026-08-02 — `audits/SPIKE-r165-mp1-merge-2026-08-02.md`; no layout was touched.** **M1: "the layout" is not one thing** — demo-felhom ships `mp0 200G / mp1 50G`, demo-hp `50G / 20G`, the golden `16G / 8G`; any plan expressed as a fixed pair is already wrong for one of the two boxes (this also invalidates §7.5's fleet-wide bound → **R-175**). **M2: the majority of `mp1` is NOT Tier-1** — on demo-felhom 1.7 G of 2.0 G is Tier-2 mirrors, plus the `userdata/import` drop zone which is on the system drive by CONTRACT (R-75); a plan accounting only for the units is wrong. Observed fill is 5% / 1% — the constraint is a ceiling problem, not a current-fill one. **M3: three assertions break, and the golden fails closed on the split in FOUR places** (`build-golden.sh:126,130` separate-mount + `:315,319` vzdump-exclusion guards), not one; the archive scope `rootfs+mp0+mp1` stays complete after the merge; `mountParity` holds for new archives. **M4 — THE BULKHEAD IS THE REAL COST:** today an overflow is refused per app with the last good unit byte-identical AND CANNOT REACH `/var/lib/docker`; after the merge it can, and a full Docker data-root is a stopped box, not a degraded one. Four candidate replacements costed (reserve / capture-path refusal / project quota / warnings-only); **not chosen — operator's ruling.** **M5: D-a's condition (1) is currently SATISFIED** — no external box is in the hub's host register (only the two ONLINE demo boxes, both **Tier 0 and reinstallable**, so migration cost for the measurable population is ZERO). **`peti-felhom` exists as a customer with NO host in the register, so its layout is UNMEASURED** and was not contacted (D-d). **Recommendation: S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), as a fresh-install shape with the demo boxes REINSTALLED.** **Two prerequisites are unmeasured → R-176.** **STOPPED for the operator's ruling; the merge is a supervised session.** | CC |
| **R-166** | ~~**App state gets a desired/observed model with its own store.**~~ Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5) | **SHIPPED + PROVEN-LIVE** (controller v0.189.0, 2026-08-02) | — | **Both blocking facts were established at source before any code was written, and the answers changed the shape.** **(a) Does a crash-safe journal already exist for the in-flight case?** YES, twice — `internal/quiesce/quiesce.go` (marker + `Recover`, proven on live hardware by Campaign 8 fault 10) and `internal/stacks/migrate.go` (`migration.json` + `RecoverMigration`) — but **neither covers the app-data path**: `DumpAppVolumesSafe` stopped and restarted an app with **no marker, no journal and not even a `defer`**. So the pattern existed and the coverage did not; `backup.AppStopGuard` copies the proven shape into its **own** file (one file, one writer). **(b) Is the SQLite store reachable?** Irrelevant, and deliberately unused: `metrics.db` is optional by design (the controller runs with it absent), and operational state must not live in a store designed to be droppable. **Shipped:** tri-state `desired_state` in `app.yaml` written ONLY by the customer's action (API action switch, `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, `.fab` import — a 14-caller census established that `StartStack`/`StopStack` must NOT be writers); `isBootOrphan` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN, never running**, so a legacy `app.yaml` keeps byte-identical pre-v0.189.0 behaviour; running-only backfill. D-b's every-container requirement was already met by `aggregateState` and was NOT re-implemented. **Live on 9201:** all three flows (stop survives a restart; a zero-container `running` app is recovered by name; a legacy app.yaml is skipped and never inferred as stopped). **Also fixed en route:** `SaveAppConfig` rebuilt `AppConfig` field-by-field (the R-100 shape) and would have dropped the new field on every save across nine call sites | — |
| **R-167** | **Storage monitoring and backup alerts.** Operator decision **D-c**, 2026-08-02 (`CONTEXT.md` S-5). Two pieces: a warning to the **customer** as a drive fills, **before** anything fails; and a failure alert to the **hub/operator** when any backup fails for **any** reason, lack of space included | **READY (M) — NEW 2026-08-02** | — | **Prerequisite for R-165** — D-a removes a wall that currently fails safely, so this ships in the same step, not after. **Subsumes and widens R-158**, which is the same alert one tier down (`NotifyBackupFailed` exists and the hub allowlists `backup_failed`; only the off-box/NAS leg calls it, `cmd/controller/main.go:659`). R-158 stays as the named seam and its proposed `unitNotify` shape is the concrete first leg — **do not file a second row for the same wire.** The customer half is new: nothing today warns before a fill, and R-158's measurement is that when it happens exactly one page says so | CC |
| **R-167** | ~~**Storage monitoring and backup alerts.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02) | — | **Operator decision D-c. It shipped BEFORE the R-165 merge, not with it** — D-a's condition (2) says the monitoring lands in the same step and never after, and landing it first is strictly better and costs nothing. **Customer half:** new `internal/fillwatch`, per FILESYSTEM (never per app — one full disk holding ten apps would fire ten times). **It emits the PRE-EXISTING `disk_warning`/`disk_critical` pair, which was allowlisted, copy'd, in `DefaultEnabledEvents` and checkbox'd with NO PRODUCER IN ANY REPO** — a complete customer pipeline with no producer, the **sixth** *built-but-never-wired* instance here; minting a new near-duplicate type would have left it inert forever. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range — **proven live: the critical crossing fired on the FREE-BYTE term (1.7 GB) at only 91% used.** Edge-triggered on escalation, state persisted, hysteresis dead zone at 75% / 7 GiB pinned by a test; a nil usage read never warns and never clears one (§8.4). The hub's two generic `customerMessages` entries were **removed** — `FormatCustomerEmail` prefers the entry over the message, so keeping them would discard the drive label and the byte figures. **Operator half: see R-158.** **Live on 9201, all three flows:** `disk_warning` then `disk_critical` both `customer | sent` with the Hungarian rendered, exactly two events across three boots (the edge trigger held on the one between), then a silent clear that re-armed. **v0.191.1** added the once-at-startup run (Daily/Every both wait for their first tick, so a box BOOTING over the line would have stayed silent up to 24 h — the R-100 shape); **v0.191.2** added a per-run positive observable, earned when a quiet run during this session's own validation proved unreadable as evidence. Follow-ups: **R-177** (no run-now path) | — |
| **R-168** | ~~CI: no runner exists, and with trunk-based pushes CI can DETECT but not BLOCK~~ | **SHIPPED — and the alarm is DEMONSTRATED** (2026-08-02) | — | **Runner live**: `homelab-manifests/gitea-system/act-runner.yaml`, an unprivileged host-mode `act_runner` in `gitea-system`, one owner-scoped registration serving all four repos (measured: tasks 7-10 all claimed by `felhom-gates-runner`). `.gitea/workflows/gates.yml` in each repo runs that repo's entry point with `--fast` and nothing else; no `uses:` step anywhere. **Six probes, all answered, none STOPped** — `audits/SPIKE-ci-runner-2026-08-02.md`. The two that changed the design: **P2** (stock image has git but NO python3 → custom image `felhom-act-runner:0.1.0`, base pinned, python3 and nothing else) and **P6** (a runner that loses `/data/.runner` re-registers and leaves a dead record behind → the PVC is load-bearing, measured both ways). **P5 is the one that mattered**: a failed run produced NO mail, NO notification row and NO log line from Gitea, so the run now sends its own alarm via Resend and prints the provider's accepted id. **Proven end to end, not asserted**: a deliberately broken commit pushed with `--no-verify` → run #6 `failure` → `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. Posture shown from the live pod spec: `privileged: false`, all caps dropped, no docker socket, no hostPath, `automountServiceAccountToken: false`, sized at half Gitea's limits so it cannot crowd out the service holding every repository on the same node. **The standing limit stays true and is written into the manifest and every workflow: it DETECTS, it does not BLOCK** — making it block is → R-169 | — |
| **R-169** | **CI can only report, because there is no gate in the road.** Every felhom repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand at. R-168's runner therefore notices a broken push *after* it has landed | **WAITING-ON-OPERATOR** (a working-style decision, not a defect) | an operator ruling | Making CI *blocking* requires two things this task deliberately did NOT do, because both change how the operator works and that is not a task's call: **(a)** branch protection on `main`, and **(b)** a pull-request workflow instead of direct-to-`main` pushes. The cost is real — every change would need a PR, which for a single-operator project may be worse than the disease. **The current arrangement is two nets, and it is not nothing**: `.githooks/pre-push` REFUSES locally, and R-168's runner NOTICES when that hook was skipped or was never armed in a clone, and emails. The honest gap is the window between a `--no-verify` push landing and the operator reading the alarm. Decide only if that window ever actually costs something | operator |
documentation/backlog/ROADMAP.md
+3 -1
View File
@@ -166,7 +166,9 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
| R-170 | **The drive-backed boot gate infers a Stop from a container count** | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | `shouldRecreateOnBoot` reads `desired_state` with the same three-way table as `isBootOrphan`; absent keeps the old `hasContainers` behaviour exactly; `presentStable` untouched. Agreement pinned from both sides against one fixture table. Live: calibre-web (`running`, zero containers) recreated and immich (`stopped`) left alone in the same reboot |
| R-171 | **The boot sweep started apps whose data drive was ABSENT** — a regression from v0.189.0 | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | Reasoned from the diff, CONFIRMED on hardware first (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The write hazard was blocked only by an ACCIDENTAL filesystem permission (host-root-owned mountpoint + unprivileged guest); the false dead-app alarm was real on every box. New fail-safe `bootrecon.StartGate` seam, also covering quiesce and in-flight app-data operations (§8.2). The rule already existed on the API path (`startGatedByMissingDrive`); the sweep bypassed it |
| R-166 | **App state gets a desired/observed model with its own store** (operator decision D-b) | M | **SHIPPED + PROVEN-LIVE — controller v0.189.0, 2026-08-02** | Tri-state `desired_state` in `app.yaml`, written ONLY by the customer's action; `bootrecon` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN** so legacy boxes keep byte-identical behaviour; running-only backfill; `backup.AppStopGuard` covers every stop→work→start window in its own marker file. **The durable fix for R-157 mechanism B.** Both blocking facts answered at source first: the crash-safe journal pattern DID already exist (quiesce, migrate) but covered **none** of the app-data path, and the SQLite store was rejected as a home because `metrics.db` is optional by design. `07`/`02` architecture docs carry the desired/in-flight/observed split. **Left for R-170:** the drive-backed boot gate still uses the container-count inference |
| R-158 | **A local Tier-1 backup failure reaches no hub channel** — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, only the off-box leg calls it | S | READY — 2026-08-02 | Fifth *seam built but never wired*; **R-97's defect one tier over**. Only caller `cmd/controller/main.go:659`; the manager's seams are tier2/offbox/offbox-enlarge (`backup.go:33,37,58`), none for the unit capture. **Ranked below R-157 by measurement:** `/backups` DOES show `✗ Adatmentés sikertelen`, it persists across a second failure and clears on recovery, and `/backups/apps` shows the last good unit's true mtime — so it is a notification gap, not silent failure. **The specific fix: `/backups/apps` is the page you would check for one app, and the only one that never says.** Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md`, `audits/CAMPAIGN-10-closeout-2026-08-02.md` |
| R-158 | ~~A local Tier-1 backup failure reaches no hub channel~~ | S | **SHIPPED — controller v0.191.0 + hub v0.89.0, 2026-08-02** | Collapsed per the lifecycle rule. Closed by **R-167** (D-c's operator half) — no second row was filed for the same wire. New `unitNotify` seam fired per app from `captureAllRecoveryUnits` with the loop continuing, carrying the target filesystem's used/free bytes. **Routed to a new OPERATOR-ONLY type `recovery_unit_capture_failed`, NOT the `backup_failed` this row proposed** — that type is customer-enabled by default and would email the customer about a failure they cannot act on; decision D-c overrides the proposal. **Flips:** the new capability-map row *"A failed per-app Tier-1 backup reaches the OPERATOR"* → **PROVEN-LIVE** (`customer | … | skipped | operator_only` observed in the hub's `notification_log`, 9201, 2026-08-02) |
| R-167 | ~~Storage monitoring and backup alerts (decision D-c)~~ | M | **SHIPPED — controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02** | Collapsed per the lifecycle rule. **Landed BEFORE the R-165 merge rather than with it** — D-a's condition (2) requires the same step and never after, and first is strictly better: the warnings were proven on hardware while the wall is still standing. Customer half = `internal/fillwatch`, per filesystem, two threshold terms (85% / 5 GiB; critical 95% / 2 GiB), edge-triggered with persisted state and a 75% / 7 GiB hysteresis return. **It emits the pre-existing `disk_warning`/`disk_critical` pair, which had NO producer in any repo — the sixth *built-but-never-wired* instance**; the hub's two generic `customerMessages` entries were deleted so the dynamic Hungarian survives. Operator half = R-158. **Flips:** two new capability-map rows → **PROVEN-LIVE**. **Follow-ups:** R-177 (no run-now path), R-175 (§7.5's bound is one box's) |
| R-165 | Merge `mp1` into `mp0` — the dedicated backup partition stops existing (decision D-a) | M | **SPIKED 2026-08-02 — WAITING-ON-OPERATOR** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md`, M1-M5, **no layout touched**. Prerequisite R-167 is now SHIPPED. **Three findings the merge session must not re-derive:** (1) *"the layout" is not one thing* — demo-felhom `200G/50G`, demo-hp `50G/20G`, golden `16G/8G`, so any fixed pair is already wrong for one box (→ R-175); (2) **`mp1` is a BULKHEAD, not only a ceiling** — an overflow today cannot reach `/var/lib/docker`, and after the merge it can, which is the one place "the merge is cheap" stops being true; (3) the golden fails closed on the split in **four** places, not one. **M5: D-a's condition (1) is currently SATISFIED** — no external box is in the hub's host register, and both demo boxes are Tier 0 and reinstallable, so migration cost for the measurable population is zero. **Recommendation: S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), fresh-install shape.** **Two prerequisites unmeasured → R-176.** Does NOT close R-163 until it lands |
| R-159 | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy** | XS | **SHIPPED** (`templates/wishlist/`, 2026-08-02) — filed for the CLASS | Image declares `VOLUME /usr/src/app/data`; template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns names only for compose-declared volumes, so `DumpAppVolumes` never sees an anonymous one. **The class is open:** any image `VOLUME` at an unmounted path is silent unbacked-up storage — **`immich-server` has one today** at `/data`, empty when measured. Proposed `REUSE.md` rule: a template mounts every path in `Config.Volumes`, or says why not |
| R-160 | **gramps-web persisted three paths and wrote to none of them** | XS | **SHIPPED** (`templates/gramps-web/`, 2026-08-02) | `/app/data` appears nowhere in the image's env; the accounts DB and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths, the template three, one a phantom. **Severity above papra's:** papra loses documents the customer may hold elsewhere; gramps-web loses the artefact built inside the app, of which no other copy exists by construction |
| R-161 | **The volume-persistence gate is enforced by convention, not automatically** | M | **RULED + SHIPPED at reduced scope** (operator, 2026-08-02; `app-catalog` `fd7747d`) | Enforcement is **convention**: the catalog repo has no CI at all (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none). **R-29's record: three orphaned gates, one enforced, and only the enforced one ever stopped anything — so the ruling copied the shape that works.** **Controller-side REJECTED with a measured reason:** a check at template load can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — it would pass on the very defect it exists to catch (the property is runtime-only; see `check-volume-persistence.py`'s header). **CI REJECTED for now** — neither repo has any, no users yet. **Shipped:** `scripts/catalog_gates.py`, one entry point over all three gates, non-zero on any failure, mandated in the catalog's `CLAUDE.md` as `site_gates.py` is. **Open residue is only the automatic half**, sufficient while one person touches templates |