docs: R-181 closed, R-156 closed, R-110 + R-115 rulings recorded, R-182 filed
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
R-181 CLOSED (controller v0.193.0 + v0.193.1) and proven live on demo-hp for BOTH reserve terms. The reserve is now a per-app, per-run ADMISSION decision taken before the app's first write and covering all three write legs, and it gained a size term. The refusal's wording was not weakened; the behaviour moved so it became true, verified by sha256 tree fingerprint. R-156 CLOSED — papra's template mounts the app's own data root. Precondition re-measured rather than inherited (both boxes were wiped today). Part 4, documentation only, nothing built: - R-110 WAITING-ON-OPERATOR -> READY. Ruling: option (b), the installer's publish channel moves to a TAG. Recorded with the condition that decides whether it works at all — it must cover BOTH the /scripts/ git-sync and the nine files the installer fetches from raw/branch/main. - R-115 WAITING-ON-OPERATOR -> READY. Ruling: mechanism (b), a build-side gate refusing to deploy or vouch an unpublished version. The third instance (agent v0.120.0) would have silently downgraded both demo boxes while succeeding. R-182 NEW: the periodic status refresh has no admission scope, so a refused app re-alerts on every poll (measured: a second alert pair 13s after the run's). Pre-existing in v0.192.0; deliberately not fixed in the R-181 task. capability map: the local-backup row moves to PROVEN-LIVE in BOTH halves. ROADMAP: R-165 collapses to CLOSED; R-181 collapsed into it. 07-backup-architecture.md: the reserve's contract stated as what the code provides (S-1 — an architectural contract changed in the same session). STATUS.md trimmed 150 -> 111 lines, "What's broken" no longer holds shipped work, and the stale "After:" line (pointing at work that shipped on 2 August) is fixed.
This commit is contained in:
+36
-4
@@ -60,10 +60,42 @@ demo boxes were wiped and reinstalled from golden 0.192.0 and taken through clai
|
||||
proved the disk shape *and* the delivery route rather than one of them twice. Live shape on both:
|
||||
`mp0` at `/var/lib/felhom`, `backup=1`, **no `mp1`**; `/var/lib/docker` and `/mnt/sys_drive` both real
|
||||
mounts of its subdirectories via `/etc/fstab`; ONE `df` figure and one device id on all three paths;
|
||||
3/3 reboots each with the binds surviving every time. **What is NOT proven is B2** → **R-181**: the
|
||||
floor guards `captureAllRecoveryUnits` and not `runVolumeDumps`, which is the leg that fills the
|
||||
volume, and its refusal message's "the previous unit is untouched" was measured false. R-165 is
|
||||
therefore **IMPLEMENTED**, not PROVEN-LIVE.
|
||||
3/3 reboots each with the binds surviving every time. B2 was **not** proven on that pass → **R-181**:
|
||||
the floor guarded `captureAllRecoveryUnits` and not `runVolumeDumps`, the leg that fills the volume,
|
||||
and its refusal's "the previous unit is untouched" was measured false. **R-181 CLOSED the same day
|
||||
(controller v0.193.0 + v0.193.1), so R-165 is now PROVEN-LIVE in both halves** — see S-14.
|
||||
|
||||
**S-14 — the reserve is a per-app, per-run ADMISSION decision, not a capture check (2026-08-03, R-181;
|
||||
controller v0.193.0 + v0.193.1).** B2 as first shipped was consulted in exactly one place —
|
||||
`captureAllRecoveryUnits`, a few KB — while `RunDBDumps`' database leg and `runVolumeDumps` wrote the
|
||||
bulk into the same `backups/primary/<app>` tree, first and unguarded. The reserve was therefore
|
||||
consumed by the very write it exists to bound, and the refusal then claimed *"the previous unit is
|
||||
untouched"* about a tree the earlier leg had already rewritten (182,272 B → 2,147,666,432 B under a
|
||||
manifest that had not moved). **Sixth entry in `CLAUDE.md`'s table of shipped guarantees the code did
|
||||
not provide, and the fourth of those found on live hardware rather than by review.**
|
||||
|
||||
- **`internal/backup/admission.go` — `admitApp` is now THE gate**, and every per-app write leg calls
|
||||
it. One verdict per app per run covers all three; they share one per-app root, which is what makes
|
||||
that honest.
|
||||
- **Decided lazily at the app's first write, never once at run start** (app A's dump can put app B
|
||||
under the reserve), **never re-decided between an app's own legs** (that is the split it closes),
|
||||
and **reset per run**.
|
||||
- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided
|
||||
inside it has already bounced the app. **After** the volume-less check, which has no write to gate.
|
||||
- **Size term added:** *would THIS app's write cross the reserve?*, estimated from the app's previous
|
||||
`.sql` + `.tar`. **No history → headroom-only**, deliberately — otherwise the first backup is the
|
||||
one that can never happen.
|
||||
- **A container-based `du` was MEASURED and rejected**, not waved away: median **~355 ms/volume** over
|
||||
66 runs on demo-hp, on volumes holding tens of KB (container start-up, not the walk). Decisive on
|
||||
top: `docker run` needs the writable layer, so the instrument can fail under exactly the pressure
|
||||
the reserve handles.
|
||||
- **The wording was NOT weakened; the behaviour moved so it became true**, and it is checked by
|
||||
sha256 tree fingerprint, never by reading the log line — the log line is what lied.
|
||||
- **v0.193.1**, found by the proof run itself: a 178 KB estimate printed as `0.00 GiB`, which reads as
|
||||
*no estimate available*. Rendering moved to `humanizeBytes`; arithmetic still in GiB.
|
||||
- **New finding, deliberately not fixed here → R-182**: `GetFullStatus`'s periodic capture sweep has
|
||||
no run scope, so a refused app re-alerts on every status refresh (measured: a second identical alert
|
||||
pair 13 s after the run's). Pre-existing in v0.192.0; R-181 changed neither caller.
|
||||
|
||||
**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
|
||||
Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
|
||||
|
||||
@@ -1,358 +1,216 @@
|
||||
# REPORT — R-178: both demo boxes reinstalled from the merged golden and proven (2026-08-03)
|
||||
# REPORT — R-181 (the reserve guards the write that fills the disk) + R-156 (papra) + two operator rulings
|
||||
|
||||
**Overwritten** per the standing rule. The prior contents (R-165, the bake, 2026-08-03) have their
|
||||
durable record in `documentation/backlog/OPEN-ITEMS.md` R-165 and the per-repo CHANGELOGs.
|
||||
**Date:** 2026-08-03 · **Repos:** `felhom-controller` (v0.192.0 → **v0.193.1**), `app-catalog-felhom.eu`, `felhom.eu` (docs only — **no hub change, no hub version bump**)
|
||||
|
||||
**Runbook, not a task.** No repo got a version bump and nothing was built. One artifact was
|
||||
**published** (agent 0.120.0) on an explicit operator ruling — see §2. All code findings are filed as
|
||||
register rows, not commits, per the runbook's §7.
|
||||
## 1. Baselines — re-read on arrival, all matched §1
|
||||
|
||||
---
|
||||
|
||||
## 1. Preconditions P1–P6, each as measured
|
||||
|
||||
| # | Precondition | Measurement |
|
||||
|---|---|---|
|
||||
| **P1** | Golden sha256 matches R-178's record | **PASS** — computed from the artifact itself on demo-hp: `sha256sum` → `54e2a4c431daf580d2807b82d810be36a7c6a9f697b27094dc2912d26a43b3e0`, 649,547,835 B. Matches R-178's recorded prefix and the hub manifest's stored value exactly. Re-verified after copying to `local`: identical |
|
||||
| **P2** | A rollback golden still exists | **PASS, three depths.** Split-layout golden **0.188.0** fetchable from Gitea (`HTTP 200`, 649,310,288 B). A **local** split-layout golden sat on each box — `local:backup/vzdump-lxc-9100-2026_07_21-18_24_49.tar.zst` (demo-hp), `…2026_07_20-17_50_57…` (demo-felhom). Best: **full guest vzdumps**, three per box, newest `vzdump-lxc-9201-2026_08_03-04_54_28.tar.zst` (2,351,557,870 B, demo-hp) and `…2026_08_03-04_44_50…` (6,323,506,165 B, demo-felhom), both ~3 h old — so each box could be put back exactly as it was |
|
||||
| **P3** | Neither demo box holds anything wanted | **PASS, stated explicitly.** demo-hp: paperless-ngx (webserver/postgres/redis) + filebrowser + samba shares. demo-felhom: immich (×4), docmost (×3), calibre-web, bookstack (×2) + filebrowser + samba. Both are demo customers (`Demo HP`, `Demo Ügyfél`); no real customer data. **All of it was destroyed by the wipes and none of it was restored** — that was the point, and the operator confirmed each wipe |
|
||||
| **P4** | The colleague's box untouched | **PASS** — `peti-felhom` is Tier 2 and out of scope. No command was sent to it; it appears in the hub customer list as `DOWN`, exactly as before |
|
||||
| **P5** | Both hosts' agent is v0.120.0 | **PASS** — `felhom-agent 0.120.0` on both, before and after |
|
||||
| **P6** | The hub's host register, BEFORE | **Captured**: `demo-felhom-8363b5 / Demo Ügyfél / 0.120.0 / ONLINE / 1-of-2 guests`; `demo-hp-bb76ea / Demo HP / 0.120.0 / ONLINE / 1-of-2`; `drill-r50-0a4f9a / drill-r50 / 0.113.0 / DOWN` (the fixture, untouched throughout) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Two findings that contradicted the runbook's premise, both surfaced before any wipe
|
||||
|
||||
**(a) The golden was ALREADY vouched.** Hub log, a positive observable:
|
||||
`2026/08/03 07:23:26 [INFO] Artifact manifest set: agent=0.119.0 golden=0.192.0 min_agent="0.113.0"
|
||||
wrapper_sha=true` — roughly ten minutes before this session's first hub read, and this session had
|
||||
POSTed nothing. The configuration page confirmed `Artifacts.GoldenVersion=0.192.0`,
|
||||
`GoldenSHA256=54e2a4c4…3b3e0`. So §7's prove-then-vouch order was already spent, and Phase A's stated
|
||||
safety ("nothing is official yet, so a failure reaches nobody") was void. **Operator ruling: accept it
|
||||
and proceed**, keeping the two different supply paths. Recorded on `CONTEXT.md` S-14, now marked SPENT
|
||||
with the reason — the rule lived only in prose and nothing enforced it.
|
||||
|
||||
**(b) Agent v0.120.0 had never been published — the vouched agent was 0.119.0.**
|
||||
`GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200). Installer step 5's
|
||||
idempotent skip requires `installed == vouched` exactly, so a documented-path reinstall would have
|
||||
**downgraded both boxes** to the pre-merge 0.119.0 — and would have *succeeded* while doing it, since
|
||||
the current `step_grows` sets `SYSDATA_GROW=0` and 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on
|
||||
error) therefore never fires. The session would have proven a stack nobody ships. **Operator ruling:
|
||||
publish and vouch first.** `scripts/publish-agent.sh 0.120.0` from a clean tree at `4bb84fc3`
|
||||
(`git status --porcelain` empty, `HEAD == origin/main`): upload HTTP 201, **round-trip GET verified**,
|
||||
`AGENT_SHA256=a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`. Vouched via
|
||||
`POST /configuration/artifacts` → `303 …flash=artifacts_set`, hub log
|
||||
`Artifact manifest set: agent=0.120.0 golden=0.192.0`; the hub resolved the sha authoritatively from
|
||||
Gitea rather than trusting the submitted value. Filed as the **third instance of R-115**, not a new
|
||||
ID — R-115 is that row, and it has been WAITING-ON-OPERATOR since 2026-07-29.
|
||||
|
||||
---
|
||||
|
||||
## 3. demo-hp — the layout proof
|
||||
|
||||
**Install path.** Uninstall: `./felhom-host-install.sh --uninstall --vmid 9201` (typed-vmid
|
||||
confirmation supplied over a pty). Install:
|
||||
|
||||
```
|
||||
./felhom-host-install.sh --customer-id demo-hp --mode appliance --vmid 9201 \
|
||||
--cores 7 --memory 26906 \
|
||||
--golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst \
|
||||
--passphrase-file /root/.pp-demo-hp
|
||||
```
|
||||
|
||||
Script fetched from `https://felhom.eu/scripts/felhom-host-install.sh`, **v1.22.0**, sha256
|
||||
`ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4` — byte-identical to the repo copy
|
||||
that was read. The passphrase went file→file into a 0600 file and never onto a command line. A
|
||||
`--dry-run` preceded the real run and resolved `manifest: agent v0.120.0 (sha a7763d31…), golden
|
||||
v0.192.0` with `grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)`.
|
||||
|
||||
**One deviation, mine, and it cost a restart.** The first attempt staged the golden on
|
||||
`felhom-backup` with `--archive-storage felhom-backup`. Pre-flight passed; **step 8/8** failed:
|
||||
`HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)` —
|
||||
the default pool-scoped ACL grants `local local-lvm felhom-pbs` only. Fixed by copying the golden to
|
||||
`local` (sha re-verified after the copy) and `--resume`. Filed as **R-180**: the condition is
|
||||
statically checkable in pre-flight, and the failure lands *after* step 4b has rotated and vaulted
|
||||
root@pam.
|
||||
|
||||
**Layout evidence.**
|
||||
|
||||
```
|
||||
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G # and NO mp1 line
|
||||
rootfs: local-lvm:vm-9201-disk-0,size=32G
|
||||
|
||||
/var/lib/felhom /dev/mapper/pve-vm--9201--disk--1 ext4 rw,relatime,stripe=16
|
||||
/var/lib/docker /dev/mapper/pve-vm--9201--disk--1[/docker] ext4
|
||||
/mnt/sys_drive /dev/mapper/pve-vm--9201--disk--1[/sys_drive] ext4
|
||||
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/felhom
|
||||
/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/docker
|
||||
/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /mnt/sys_drive
|
||||
|
||||
stat -c %d → 64519 for all three paths
|
||||
/etc/fstab: /var/lib/felhom/docker → /var/lib/docker ; /var/lib/felhom/sys_drive → /mnt/sys_drive
|
||||
both binds writable (touch succeeded on each)
|
||||
```
|
||||
|
||||
`/mnt/sys_drive` shows a second `findmnt` row — it is the controller container's
|
||||
`-v /mnt:/mnt:rslave` propagation (peer group 383 vs the fstab bind's 333), the same shape the split
|
||||
layout had, not a stacked bind.
|
||||
|
||||
**Reboots — each individually, minimum three:**
|
||||
|
||||
| # | started | controller healthy | outcome |
|
||||
| Repo | `main` @ arrival | Version | Shipped |
|
||||
|---|---|---|---|
|
||||
| 1 | 08:13:33 | 08:13:55, `Up 12 seconds (healthy)` | one filesystem, 69G/65G on both paths |
|
||||
| 2 | 08:13:59 | 08:14:15, `Up 7 seconds (healthy)` | same |
|
||||
| 3 | 08:14:19 | 08:14:35, `Up 7 seconds (healthy)` | same |
|
||||
| `felhom-controller` | `4be6467b501b` | v0.192.0 | **v0.193.0 `fef07c3`** → **v0.193.1 `6c43bf6`** |
|
||||
| `app-catalog-felhom.eu` | `7cb58ecdf8e7` | n/a | `122bbee` |
|
||||
| `felhom.eu` | `6b5d64c1fa73` | hub v0.89.0 | docs only, **no bump** |
|
||||
|
||||
After all three, `mountpoint -q` returns true for **all three paths** and both binds still resolve to
|
||||
the single volume's subdirectories — which is what the reboots exist to test. `uptime -s` =
|
||||
`2026-08-03 06:14:24 UTC`, matching reboot 3, so these were real reboots.
|
||||
All three clean (`git status --porcelain` empty, `HEAD == origin/main`) before every build.
|
||||
|
||||
**The journey — method stated: endpoint-level, not a browser.** `claude-in-chrome` does not exist on
|
||||
DooPlex; every step below invoked the exact endpoint the dashboard's own JavaScript calls.
|
||||
## 2. The fix
|
||||
|
||||
| Leg | Endpoint | Observable |
|
||||
|---|---|---|
|
||||
| Claim | `POST /claim` with the pre-auth HMAC CSRF token (64 chars) **and** its `felhom_claim_csrf` cookie | `302 → /`; gate discriminator flipped `{"error":"dashboard not yet claimed"}` → `{"error":"authentication required"}`. **The code is emailed-only (R-119) — the operator supplied it**, after an operator resend rotated the generation (the previous code had been consumed 12 d earlier) |
|
||||
| Deploy | `POST /api/stacks/<n>/deploy` with `{"values":{…}}`, field values scraped from the deploy form exactly as the form's own `fetch` does | `{"ok":true}`; `privatebin Up (healthy)`, then `opengist Up (healthy)` |
|
||||
| Back up | `POST /api/debug/backup/dbdump` — this runs the **production** `RunDBDumps` path (DB leg → `runVolumeDumps` → `captureAllRecoveryUnits`); the debug route only starts it instead of waiting for 03:30 | `Volume dump: opengist/… → 178.0 KB`, `privatebin/… → 2.5 KB`, `App-data backup completed … 2 volume dump(s) (3.962s)`, `Recovery unit captured for …` ×2 |
|
||||
| **Restore** | `POST /backup/restore` `stack_name=privatebin snapshot_id=primary` | A marker planted in the live volume was **deleted**, then restored: `{"ok":true,"message":"privatebin visszaállítva (primary)."}` in **9.2 s**, and the marker returned with an **identical sha256 `ac1faae6134871d640d5cb1bbc6b5092d1e96b7e46c6c3bcb6e8236cc33ae861`**. App `Up (healthy)` afterwards |
|
||||
**One admission verdict per app per run** (`controller/internal/backup/admission.go`), taken before
|
||||
that app's **first** write and consulted by all three legs — DB dump, volume dump, unit capture. The
|
||||
three write under one per-app root (`appbackup.RecoveryUnitPath`), which is what makes one verdict
|
||||
able to cover them honestly.
|
||||
|
||||
**Recovery unit path, and it lands on the single volume:**
|
||||
`/mnt/sys_drive/felhom-data/backups/primary/<app>/{manifest.json,compose/,volume-dumps/}`, whose `df`
|
||||
is `/dev/mapper/pve-vm--9201--disk--1` — the merged volume.
|
||||
- **Lazy, not run-wide.** App A's dump can put app B under the reserve; a run-start verdict reads a
|
||||
disk that no longer exists. **Never re-decided between an app's own legs** — that is the split being
|
||||
closed. **Reset per run.**
|
||||
- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act, so a refused app is never
|
||||
bounced. **After** the volume-less check, which has no write to gate.
|
||||
- **Exactly one operator alert per refused app per run.** Leg order unchanged.
|
||||
- **Size term added:** *would this app's write cross the reserve?* — estimated from its previous
|
||||
`.sql` + `.tar`. **No history → headroom-only**, or the first backup becomes the one that can never
|
||||
happen; the alert says so when that applies.
|
||||
|
||||
**Note on the first app chosen.** `privatebin`'s catalog entry declares no `backup:` section, so its
|
||||
first per-stack capture produced `"volume_dumps": null` — correct for that declaration, not a defect,
|
||||
but it means a per-stack `POST /stacks/<n>/backup` writes compose+config only; the volume leg lives in
|
||||
the full pass. A second app (`opengist`) was deployed so the run had both a capture and, later, a
|
||||
refusal.
|
||||
## 3. Files
|
||||
|
||||
**The ceiling is gone, measured:** a recovery unit can use **65 GiB** — the whole volume — against the
|
||||
**19 GiB** the pre-wipe `mp1` slice offered (`/dev/mapper/pve-vm--9201--disk--2 20G 95M 19G 1%`).
|
||||
|
||||
---
|
||||
|
||||
## 4. The floor's first live firing — and it does not do what it says
|
||||
|
||||
**Instrument, proven before use.** demo-hp's thin pool is **53.93 GiB** and the auto-sized volume is
|
||||
70 G, so a real fill to 97 % would have exhausted the pool and corrupted every guest on the box,
|
||||
including the protected `drill-r50` fixture. A 5 GiB `fallocate` probe moved guest `df` from
|
||||
`977M used / 65G avail` to `6.0G / 60G` while thin-pool `data_percent` stayed **29.03 → 29.03** —
|
||||
zero blocks allocated — and cleanup returned both to baseline. The floor reads `statfs`, which is
|
||||
exactly what `fallocate` moves, so the condition it guards is genuinely present.
|
||||
|
||||
**Setup.** A **real** 2 GiB file in opengist's data volume (so its capture writes 2 GiB), then
|
||||
`fallocate` to leave `69G / 63G used / 3.0G avail / 96%` — **both floor terms deliberately still
|
||||
clear**, so the run would start. Capture order was established empirically from the previous run's log
|
||||
(opengist first, privatebin second), not assumed.
|
||||
|
||||
**What happened, 06:40:03:**
|
||||
|
||||
```
|
||||
Volume dump: opengist/opengist_opengist_data → 2.0 GB # unguarded
|
||||
Volume dump: privatebin/privatebin_privatebin_data → 2.5 KB
|
||||
App-data backup completed … 2 volume dump(s) (20.737s)
|
||||
[WARN] Recovery unit capture REFUSED for opengist — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted
|
||||
[WARN] Recovery unit capture REFUSED for privatebin — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted
|
||||
[INFO] Event pushed: recovery_unit_capture_failed (error) — … ×2 (HTTP 200)
|
||||
```
|
||||
|
||||
**What holds:** it refuses **per app** rather than aborting the run; **nothing was deleted** (both
|
||||
units present afterwards); and the operator alert reached the hub — `recovery_unit_capture_failed`,
|
||||
severity `error`, accepted `HTTP 200`, twice.
|
||||
|
||||
**What does not hold — measured, not inferred.** The refusal message claims *"the previous unit is
|
||||
untouched"*:
|
||||
|
||||
| unit file | before | after |
|
||||
|---|---|---|
|
||||
| `privatebin/volume-dumps/privatebin_privatebin_data.tar` | `26c546c2…` | **`b538ab89…`** |
|
||||
| `opengist/volume-dumps/opengist_opengist_data.tar` | 182,272 B | **2,147,666,432 B** |
|
||||
|
||||
Both were rewritten by the earlier leg, while each `manifest.json` kept
|
||||
`created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so
|
||||
a unit's payload can be swapped under a stale descriptor and nothing inside the unit can detect it.
|
||||
|
||||
**Cause, in the code and not the log.** The floor is consulted in exactly one place —
|
||||
`m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which
|
||||
writes a manifest and a compose copy: a few KB. `runVolumeDumps` (`backup.go:535`) — the leg that
|
||||
writes the bulk, and the leg that consumed the reserve — has **no floor check at all**; its gates are
|
||||
protected-stack, volume-less, disconnected, decommissioned. And it runs first *by design*
|
||||
(`backup.go:483`). **The floor guards the cheap leg and not the leg that fills the volume.**
|
||||
|
||||
This is why R-165 is IMPLEMENTED and not PROVEN-LIVE: that row records B2 as the deliberate
|
||||
replacement for the bulkhead the `mp1` partition provided, and pre-merge the unguarded leg could only
|
||||
fill a dedicated 20 G partition — post-merge it can reach Docker's data-root. Filed as **R-181**.
|
||||
**No code was written**, per the runbook's §7.
|
||||
|
||||
**Cleanup:** fill file and payload removed; `df` back to `1.2G used / 65G avail`; a clean re-run left
|
||||
both units valid (`Volume dump … 178.0 KB` / `2.5 KB`, `App-data backup completed … (2.399s)`).
|
||||
|
||||
---
|
||||
|
||||
## 5. demo-felhom — the pipeline proof
|
||||
|
||||
**Install path — deliberately different, and this is the reason the second box exists.**
|
||||
|
||||
```
|
||||
./felhom-host-install.sh --customer-id demo-felhom --mode appliance --vmid 9201 \
|
||||
--cores 3 --memory 12288 \
|
||||
--force-gitea-golden \
|
||||
--passphrase-file /root/.pp-demo-felhom
|
||||
```
|
||||
|
||||
A copy of golden 0.192.0 already sat on this box's `local` storage — it is where the golden was
|
||||
**baked** at 06:58 (a `.log` beside it), sha `54e2a4c4…`. `--force-gitea-golden` is the documented
|
||||
C.3 customer path and overrides local discovery in **both** pre-flight and step 7, which pre-flight
|
||||
confirmed: `golden: none local — will fetch + verify from Gitea in step 7/8`. The bake artifact was
|
||||
left untouched.
|
||||
|
||||
**`fetch_verify` succeeding against the vouched sha — the observable this box exists to produce:**
|
||||
|
||||
```
|
||||
5/8 fetching agent binary v0.120.0 from Gitea …
|
||||
verified sha256 a7763d31b55b5ce7… matches the hub manifest
|
||||
7/8 fetching golden v0.192.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
|
||||
verified sha256 54e2a4c431daf580… matches the hub manifest
|
||||
golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
|
||||
Day-0 provision SUCCESS — vmid=9201 host_id=demo-felhom-8363b5 customer=demo-felhom
|
||||
```
|
||||
|
||||
Both artifacts — agent and golden — were fetched anonymously (the normal customer shape) and
|
||||
sha-verified against the manifest. Controller `0.192.0` healthy.
|
||||
|
||||
**Layout:** `mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=250G`;
|
||||
`pct config 9201 | grep -c '^mp1:'` → **0**. Both binds real mounts
|
||||
(`…disk--1[/docker]`, `…disk--1[/sys_drive]`), `df` one figure — `246G 977M 233G 1%` on all three
|
||||
paths — `stat -c %d` = `64519` on all three, fstab carries both binds.
|
||||
|
||||
**Reboots:** 1 — 09:18:55 → healthy 09:19:09; 2 — 09:19:12 → 09:19:26; 3 — 09:19:30 → 09:19:45. All
|
||||
three paths still mountpoints after each; `uptime -s` = `2026-08-03 07:19:34 UTC`, matching reboot 3.
|
||||
|
||||
**Journey:** claim (`302 → /`, discriminator flipped to `authentication required`; code supplied by
|
||||
the operator after a resend) → deploy `opengist` (`{"ok":true}`, `Up (healthy)`) → capture
|
||||
(`Volume dump: opengist/… → 178.0 KB`, `Recovery unit captured`, unit on
|
||||
`/dev/mapper/pve-vm--9201--disk--1`, marker present inside the tar) → **restore**
|
||||
(`{"ok":true,"message":"opengist visszaállítva (primary)."}` in **9.4 s**, marker back with identical
|
||||
sha256 `bc5507987f3f56dc19a9c24785826c304f1a986998d4aeb0b06d1a810b59e939`, app healthy).
|
||||
|
||||
**Ceiling:** 233 GiB available to a recovery unit, against the 45 GiB the pre-wipe `mp1` offered.
|
||||
|
||||
**Step 8 was NOT repeated here, deliberately** — stating it rather than leaving it ambiguous. The
|
||||
floor fired on demo-hp and R-181 characterises it fully; re-firing would add no information and would
|
||||
mean filling a 246 G volume.
|
||||
|
||||
---
|
||||
|
||||
## 6. Vouching
|
||||
|
||||
Already done before the session (§2a), at **07:23:26 CEST on 2026-08-03**, by an operator action in
|
||||
the hub UI. The session's own manifest write was the **agent** half, at **07:44:31**:
|
||||
`Artifact manifest set: agent=0.120.0 golden=0.192.0 min_agent="0.113.0" wrapper_sha=true`. The
|
||||
manifest afterwards, read back: `agent_sha256=a7763d31b55b5ce7…10b9d`,
|
||||
`golden_sha256=54e2a4c431daf580…43b3e0`, `min_agent=0.113.0`, `wrapper_sha256=104db0a4…` (preserved
|
||||
verbatim). The R-120 gate did not block: golden 0.192.0 equals the newest controller the fleet
|
||||
reports.
|
||||
|
||||
---
|
||||
|
||||
## 7. Teardown — all three layers
|
||||
|
||||
1. **The machine.** No throwaway guest was created this session, so there is none to delete. VM 300
|
||||
`drill-r50` on demo-hp — the protected drift fixture — was never touched and is still `stopped`;
|
||||
it is not in the `felhom` pool (`pvesh get /pools/felhom` listed only `lxc/9201`), so the
|
||||
uninstall's shared-box logic never reached it.
|
||||
2. **The host.** `local-lvm`, before → after: **demo-felhom 29.31 % → 1.37 %** (the old 200 G + 50 G
|
||||
volumes returned; the new 250 G volume is thin and barely allocated). **demo-hp 39.13 % →
|
||||
36.75 %** — *higher than a clean reinstall would leave it*, because the floor test's 2 GB tar
|
||||
blocks cannot be reclaimed: `fstrim` inside an unprivileged LXC returns
|
||||
`FITRIM ioctl failed: Operation not permitted`. No operational impact (the guest shows 65 G free of
|
||||
69 G, the host 35.7 GiB free of 53.93), but it is real residue and is stated rather than rounded
|
||||
away. Each box's data volume is now the only data volume; no old guest volumes remain.
|
||||
**Residue found and cleared by hand on demo-hp:** `--uninstall` left the NAS network-storage units
|
||||
`mnt-felhom\x2ddrives-Felhom\x2dShare.{mount,automount}` behind (automount `failed`, parent bind
|
||||
still mounted) → **R-179**. demo-felhom left none, because it had no network share configured.
|
||||
3. **The hub.** **No old records exist to dispose of, and this is the honest finding, not an
|
||||
omission:** both enrollments were **idempotent** — `host REUSED (idempotent — existing
|
||||
credential)`, `host_id: demo-hp-bb76ea` and `demo-felhom-8363b5`, the same ids as before. The
|
||||
reinstalls therefore produced **no new host records**, so nothing was orphaned and nothing needed
|
||||
deleting. Final register: `demo-felhom-8363b5 ONLINE 0.120.0`, `demo-hp-bb76ea ONLINE 0.120.0`,
|
||||
`drill-r50-0a4f9a DOWN 0.113.0` — the same three rows as at P6. No scratch customers were created.
|
||||
`/appliances` returns 404 on hub 0.89.0 — there is no appliance-record surface to clean.
|
||||
|
||||
**Secrets:** both retrieval passphrases were moved file→file into 0600 files, used via
|
||||
`--passphrase-file`, and `shred -u`'d afterwards on both hosts along with the session cookie files and
|
||||
helper scripts; the local scratch copies were deleted. Nothing was written to a committed file and
|
||||
`curl -w '%{redirect_url}'` was never used (R-132).
|
||||
|
||||
---
|
||||
|
||||
## 8. Registers changed
|
||||
|
||||
| Row | Change |
|
||||
| File | |
|
||||
|---|---|
|
||||
| **R-178** | **CLOSED** — both boxes reinstalled and proven, by two different supply paths |
|
||||
| **R-165** | **IMPLEMENTED**, not PROVEN-LIVE — operator ruling; the layout half is proven, the B2 half is not (→ R-181) |
|
||||
| **R-115** | **Third instance recorded** — agent 0.120.0 built, deployed to both hosts, never published |
|
||||
| **R-181** *(new)* | The capture floor guards the recovery-unit leg and not `runVolumeDumps`; its "previous unit is untouched" claim measured false |
|
||||
| **R-180** *(new)* | `--archive-storage` is not cross-checked against the ACL grant; the 403 lands at step 8/8, after root@pam has been rotated |
|
||||
| **R-179** *(new)* | `--uninstall` leaves NAS network-storage systemd units behind when a share was configured |
|
||||
| `controller/internal/backup/admission.go` | **new** — the gate, the memo, the estimator |
|
||||
| `controller/internal/backup/admission_test.go` | **new** — 11 tests |
|
||||
| `controller/internal/backup/backup.go` | run scope + gates in the DB and volume legs |
|
||||
| `controller/internal/backup/recovery_unit.go` | `floorVerdict` size-aware; capture leg via `admitApp` |
|
||||
| `controller/internal/backup/capture_floor_test.go` | 3 call sites updated for the new signature |
|
||||
| `controller/README.md`, `REUSE.md`, `CHANGELOG.md` | |
|
||||
| `app-catalog-felhom.eu/templates/papra/docker-compose.yml` | mount moved to `/app/app-data` |
|
||||
|
||||
**IDs established free before minting:** `grep -ro "R-179\b\|R-180\b\|R-181\b\|R-182\b"` over
|
||||
`documentation/` and `*.md` → **0 hits**, and over all four repo roots (`felhom-agent`,
|
||||
`felhom-controller`, `felhom.eu`, `app-catalog-felhom.eu`) → **0 hits**. `R-182` was checked and left
|
||||
unused.
|
||||
## 4. Tests — 28 packages `ok`, `rc=0` (read separately from any commit)
|
||||
|
||||
---
|
||||
All 11 new tests pass, plus the pre-existing floor suite. Refusal assertions are **sha256 tree
|
||||
fingerprints before and after**, never log lines — the defect being fixed *is* a log line the tree
|
||||
contradicted.
|
||||
|
||||
## 9. CI — checked by run id, not assumed
|
||||
The DB leg cannot run without Docker (`DiscoverDatabases` shells out), so its gate is pinned by an
|
||||
**AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne`. `strings.Contains` is
|
||||
insufficient: a commented-out call still contains the string.
|
||||
|
||||
Two commits, both docs-only.
|
||||
### Red-proofs — each demonstrated failing, then restored
|
||||
|
||||
| repo | commit | CI run | result |
|
||||
|---|---|---|---|
|
||||
| `felhom-agent` | `9dfd89c` — *docs: agent 0.120.0 published + vouched…* | **run 46**, `head_sha 9dfd89cb` | **success** (`gates`) |
|
||||
| `felhom.eu` | `aa62449` — *R-178 CLOSED: both demo boxes reinstalled…* | **run 47**, `head_sha aa624496` | **success** (`gates`) |
|
||||
| # | Mutation | Result |
|
||||
|---|---|---|
|
||||
| 1 | **Both** dump-leg `admitApp` gates removed (= exactly v0.192.0) | Scenario A **RED** — *"the VOLUME leg ran for a refused app"*; with the leg assertions temporarily made non-fatal, the **tree fingerprint changed** too. Also red: Scenario C, Scenario D, and the AST wiring test (which named the DB leg specifically) |
|
||||
| 2 | The entire size term removed from `floorVerdict` (both its thresholds) | Scenario D **RED** — 0 alerts where 1 was required |
|
||||
| 3a | The reserve removed entirely | Scenario F **PASSED — recorded honestly.** The specified mutation does not exercise the assertion: removing the reserve makes every app write, which overwrites and adds but **deletes nothing**, so a deletion-watching test correctly stays green |
|
||||
| 3b | A prune injected into the refusal path | Scenario F **RED** — this is the mutation that proves the test watches deletion |
|
||||
| 4 | Floor moved above the warning band (90% / 6 GiB) | `TestFloorSitsBelowTheCriticalWarningBand` **RED** |
|
||||
|
||||
Queried with
|
||||
`curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/<repo>/actions/tasks?limit=3"` and matched on
|
||||
`head_sha`, per `CLAUDE.md`'s pull-check rule — CI mails on failure, which is a push signal; this is
|
||||
the pull check that catches a lost or unread mail.
|
||||
Every mutation removed **every** guard its test covers (#1 removed both dump-leg gates, not one).
|
||||
|
||||
**`--no-verify` was NOT used**, anywhere. This clone is armed (`core.hooksPath = .githooks`). Both
|
||||
gate entry points were also run by hand before committing:
|
||||
`felhom.eu/scripts/repo_gates.py` → **all five OK** (site, hostinstall, hub-confirm,
|
||||
manifest-bearer, reuse-refs), and `felhom-agent/scripts/agent_gates.py` → **OK** (reuse-refs).
|
||||
No test suite was run because no code changed in either repo.
|
||||
## 5. Live validation — demo-hp guest 9201 (Tier 0), the method that found the defect
|
||||
|
||||
---
|
||||
**Method:** endpoint-level — `POST /api/debug/backup/dbdump`, the exact endpoint the debug UI button
|
||||
calls, which runs the production `RunDBDumps`. No browser on DooPlex.
|
||||
|
||||
## 10. Observations — noticed and NOT acted on
|
||||
**The instrument was re-proven before use.** demo-hp's thin pool is 53.93 GiB, so a real fill of a
|
||||
70 G volume would exhaust it and corrupt every guest. A 5 GiB `fallocate` step moved guest `df`
|
||||
1.2G → 6.2G while thin-pool `data_percent` held **36.83 → 36.83** — zero blocks allocated. Re-checked
|
||||
at every step of the fill.
|
||||
|
||||
- **The runbook's central claim was wrong in a way that mattered.** R-178 said *"a reinstall is now a
|
||||
self-contained piece of work with no code left to write."* True about code; false about the artifact
|
||||
channel — the agent half of the merge was unpublished, and following the documented path without
|
||||
checking would have downgraded both boxes and produced a green, meaningless result. **No code change
|
||||
was needed to make any step pass** — §10 asks this loudly, and the answer is no. What was needed was
|
||||
a publish.
|
||||
- **Prove-then-vouch was already spent when the session opened.** Not a defect in anything, but the
|
||||
rule protected nothing because nothing enforced it. R-120's gate is the shape that would.
|
||||
- **A per-stack `POST /stacks/<n>/backup` does not produce volume dumps** — the volume leg lives in the
|
||||
full app-data pass. Not wrong, but the endpoint's name suggests otherwise and it cost time here.
|
||||
- **privatebin's recovery unit carries no user data** (its catalog entry declares no `backup:`
|
||||
section), so restoring it loses every paste. That may be intentional for an expiring, E2E-encrypted
|
||||
paste bin — but nothing in the app's description tells the customer so. Catalog question, not filed.
|
||||
- **V-c doubles systemd's mount-unit count** — every docker overlay appears twice,
|
||||
`var-lib-docker-…-merged.mount` and `var-lib-felhom-docker-…-merged.mount`, because `/var/lib/docker`
|
||||
is a bind of `/var/lib/felhom/docker`. Cosmetic, inherent to the chosen variant, no action.
|
||||
- **`fstrim` cannot run inside the guest** (`EPERM`, unprivileged LXC), so space freed inside the guest
|
||||
is not returned to the thin pool. It did not matter here; on a box that fills and empties repeatedly
|
||||
it would.
|
||||
- **The floor's own message mixes units** — it reports `64.2/68.7 GB used (93%)` against a threshold
|
||||
stated as `97% used or 1.0 GiB free`, while `df` showed 96 %. GB-vs-GiB, so the percentage term
|
||||
fires later than an operator reading `df` would expect. Minor; noted on R-181's fix shape rather
|
||||
than filed separately.
|
||||
### Headroom term — 08:59:46, 906 MB free / 99% used
|
||||
|
||||
| Observable | Result |
|
||||
|---|---|
|
||||
| Tree fingerprint before | `TREE_SHA=111d1760c18d3440f700634ab325f8b8` (10 files; opengist's tar **182,272 B** — R-181's own "before" figure) |
|
||||
| Tree fingerprint after | **`111d1760c18d3440f700634ab325f8b8` — identical** |
|
||||
| Volume dumps written | **0** (baseline run at 08:58 wrote 2) |
|
||||
| `Stopping <app> for safe volume dump` | **absent** — and this is evidence, not an absence, because that line **is** present in the 08:58 baseline |
|
||||
| Operator alerts | one `recovery_unit_capture_failed` per app, severity `error`, HTTP 200 |
|
||||
|
||||
Free space restored → re-run at **09:01:33**: both apps captured normally.
|
||||
|
||||
### Size term — 09:03:00, proven separately
|
||||
|
||||
Reproducing the original sequence: a real 2 GiB file planted in opengist's volume, backed up so its
|
||||
**previous** tar became **2,147,666,432 B** (the exact live figure), then the filesystem set to
|
||||
**91% used / 2.9 GB free — both headroom terms deliberately clear**.
|
||||
|
||||
- **opengist refused `(size)`** — *"this app's last backup was 2.0 GB and writing it again would cross the reserve"*
|
||||
- **privatebin ADMITTED and dumped normally** — the term is per-app, not a global halt
|
||||
- Tree unchanged; 1 volume dump instead of 2
|
||||
|
||||
### One honest correction to the "app not stopped" claim
|
||||
|
||||
`StartedAt` on both apps *did* move, 26 s **after** the refusal. It was the **quiesce loop** for the
|
||||
whole-guest PBS backup, which my fill had broken — not the app-data path. Its own backoff logic then
|
||||
behaved correctly (*"deferring its next quiesce by 15m so the apps are not stopped again for a backup
|
||||
that cannot succeed"*). The app-data claim rests on the **absence of the `Stopping … for safe volume
|
||||
dump` line**, which is the line that appears when that leg bounces an app.
|
||||
|
||||
## 6. The `du` measurement (§Part 1.3) — measured, then rejected
|
||||
|
||||
**66 timed runs** on demo-hp guest 9201, `docker run --rm -v <vol>:/v alpine du -sb /v`:
|
||||
**median ~355 ms per volume, range 341–404 ms** — on volumes holding **tens of KB**. The cost is
|
||||
container start-up, not the walk, so it does not shrink for small apps and only grows for real ones.
|
||||
|
||||
**Rejected**, on two grounds beyond the number: `docker run` needs the writable layer, so the
|
||||
measurement mechanism can fail under exactly the disk pressure the reserve exists to handle; and the
|
||||
previous-dump estimate measures the **artifact that will be written** rather than the live volume,
|
||||
which is the truer predictor. The previous-dump estimate stands.
|
||||
|
||||
## 7. The refusal message as shipped, and what it guarantees
|
||||
|
||||
```
|
||||
[WARN] [backup] App backup REFUSED for opengist (headroom) — refused: backing up this app would
|
||||
leave the filesystem below the reserve (reserve: 97% used or 1.0 GiB free; the filesystem is already
|
||||
below it, before this app's estimated 178.0 KB write) — /mnt/sys_drive: 64.3/68.7 GB used (94%),
|
||||
0.9 GB free; NO database dump, NO volume dump and NO recovery-unit capture was written for it, the
|
||||
previous unit is untouched and NOTHING was deleted
|
||||
```
|
||||
|
||||
**It guarantees, for that app in that run:** no DB dump, no volume dump and no capture were written;
|
||||
every file under `backups/primary/<app>` is byte-identical; the app was not stopped; nothing anywhere
|
||||
was deleted; exactly one operator alert was sent. All five verified by fingerprint above.
|
||||
|
||||
**The wording was not weakened to fit the behaviour** — the behaviour moved so the wording became
|
||||
true. What was *added* is the bound term (`headroom` / `size`) and the estimate.
|
||||
|
||||
**v0.193.1 — found by this very proof run.** The estimate was rendered fixed to two-decimal GiB, so
|
||||
opengist's real **178 KB** printed as `estimated 0.00 GiB write`, which reads as *no estimate was
|
||||
available* — the opposite of what happened. Shipped the same session because it is the same defect
|
||||
class the whole task is about. Re-verified live after redeploy: `estimated 178.0 KB write`.
|
||||
|
||||
## 8. papra (R-156, last leg)
|
||||
|
||||
**Precondition checked, not inherited** — both boxes were wiped and rebuilt today, so the 2 August
|
||||
evidence was re-measured: `docker ps -a` (**including stopped**) on **both** demo guests → no papra;
|
||||
hub `/hosts` → exactly two enrolled hosts (`demo-felhom-8363b5`, `demo-hp-bb76ea`), **zero** papra.
|
||||
|
||||
**Decided from the image, not the README:** `WORKDIR=/app`, `DATABASE_URL=file:./app-data/db/db.sqlite`,
|
||||
`DOCUMENT_STORAGE_FILESYSTEM_ROOT=./app-data/documents`, `PAPRA_CONFIG_DIR=./app-data` — and
|
||||
**`/app/data` does not exist in the image at all**.
|
||||
|
||||
**Departure from the task's stated preference order, stated because it was deliberate.** Option (1)
|
||||
(reconfigure the app to write to `/app/data`) *was* available — all three paths are env-settable. Not
|
||||
taken: it enumerates data paths, so a fourth added upstream would silently escape to the writable
|
||||
layer again — this defect re-armed and invisible. Mounting the app's own data **root** captures every
|
||||
current and future path by construction.
|
||||
|
||||
**Gate output — the arbiter, run in both directions:**
|
||||
|
||||
- fixed → `papra CLEAN`, with the self-test passing on that run: *"prober flags the R-156 signature and clears a correct template — trustworthy"*
|
||||
- reverted to `/app/data` (red-proof on the **real template**, not just the canary) → `BROKEN`: *"mount /app/data is NOT writable by the app's own uid=999"*, *"DATA in the writable layer at /app/app-data/db (db_signature=True, e.g. ['db.sqlite'])"*, *"declared volume /app/data is EMPTY"*
|
||||
- `catalog_gates.py papra` (full, not `--fast`) → **rc=0**, all three gates OK
|
||||
|
||||
**Two operational findings about the gate:** it needs **root** (it reads `/var/lib/docker/volumes`,
|
||||
mode `drwx--x---`; as a normal user its own canary fails UNDETERMINED and it correctly refuses a
|
||||
verdict — fail-closed working as designed), and it hardcodes scratch path `/srv/felhom-gate`, created
|
||||
on DooPlex. Unscoped it deploys all 53 templates; that run was aborted after 10 minutes and its
|
||||
`volgate-*` scratch projects were cleaned up.
|
||||
|
||||
## 9. §3's correction — confirmed in passing, not chased
|
||||
|
||||
`restore_points.go:57-59` takes the manifest's mtime and then `newestArtifact` over the `.sql` and
|
||||
`.tar` files, so **the newest of the three wins**. The restore point does **not** show a stale
|
||||
timestamp. Confirmed and dropped, as instructed.
|
||||
|
||||
## 10. Register
|
||||
|
||||
| ID | Change |
|
||||
|---|---|
|
||||
| **R-181** | **CLOSED — SHIPPED** (v0.193.0 + v0.193.1), with the live evidence above |
|
||||
| **R-156** | **CLOSED** — all three apps fixed |
|
||||
| **R-110** | WAITING-ON-OPERATOR → **READY**, ruling attached: **option (b), tag-tracked**, and it must cover **both** channels (the `/scripts/` git-sync *and* the nine files fetched from `raw/branch/main`) or it only half-works |
|
||||
| **R-115** | WAITING-ON-OPERATOR → **READY**, ruling attached: **mechanism (b)**, a build-side gate refusing to deploy or vouch an unpublished version; the third instance (agent v0.120.0) would have silently downgraded both demo boxes while reporting success |
|
||||
| **R-182** | **NEW.** ID established free: `grep -ro "R-182\b"` over `documentation/` and `*.md` → 2 hits, both prose in `REPORT.md` recording it as *"checked and left unused"*; `R-183` → 0 hits and remains free |
|
||||
|
||||
**R-165** is collapsed to CLOSED/PROVEN-LIVE in `ROADMAP.md`; the capability map's local-backup row
|
||||
moves to **PROVEN-LIVE, both halves**, because the live fill proved the fixed behaviour for **both**
|
||||
reserve terms.
|
||||
|
||||
## 11. Observations — noticed, documented, NOT acted on
|
||||
|
||||
1. **R-182 (filed).** The periodic status refresh (`GetFullStatus` → `captureAllRecoveryUnits`) runs
|
||||
with no admission scope, so a refused app re-alerts on every poll — measured live: a second
|
||||
identical alert pair 13 s after the run's. **Pre-existing in v0.192.0**; R-181 changed neither
|
||||
caller. Its mitigation is a *comment* claiming the hub owns cooldown — which is exactly the
|
||||
"invariant asserted in a comment with no test pinning it" shape, so verify at the hub before
|
||||
scoping.
|
||||
2. **A reserve refusal does not make the run fail.** The DB and volume legs record `SKIP`, not `FAIL`,
|
||||
so `lastDBDump.Success` stays true and the customer-facing status does not turn red. Deliberate and
|
||||
consistent with v0.192.0 (the capture refusal never set it either), and the operator alert is the
|
||||
signal — but it means "backup succeeded" and "every app was backed up" are not the same statement.
|
||||
3. **`UnitSpace.UsedPercent` and `df` disagree** — `df` reported 99% where the alert said 94%, because
|
||||
`df`'s figure accounts for ext4 reserved blocks and the floor's does not. Harmless here (the
|
||||
free-byte term bound), but a percent-term threshold is being compared against a number the operator
|
||||
cannot reproduce with `df`.
|
||||
4. **The whole-guest PBS backup fails when the volume is near-full**, pushing
|
||||
`whole_guest_backup_failed` (severity `error`). Expected under a deliberate fill, and its backoff
|
||||
behaved correctly; noted because it is collateral any future fill test will also produce.
|
||||
|
||||
## 12. Teardown
|
||||
|
||||
Fill file removed; the planted 2 GiB file removed; a final backup regenerated a correct 178 KB tar;
|
||||
`pct fstrim 9201` returned 67.5 GiB and the thin pool settled at **29.43%**, *below* its 36.83%
|
||||
baseline. The backups tree is byte-identical to the pre-test fingerprint. Guest helper scripts and the
|
||||
credential file `shred`-ed. `volgate-*` scratch compose projects removed; the unrelated 9-day-old
|
||||
`jarr-*` containers on DooPlex were left untouched. papra is **not** left deployed.
|
||||
|
||||
No `--no-verify` was used on any push; the `felhom-controller` pre-push hook ran and reported
|
||||
`gates OK` on both pushes.
|
||||
|
||||
@@ -27,107 +27,74 @@ time, and an app switched off deliberately stayed off every time.
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
*(R-95, R-87)*
|
||||
|
||||
**Three apps out of fifty-three kept their data where backups never looked.** They reported healthy;
|
||||
the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is
|
||||
installed nowhere. *(R-156)*
|
||||
**A full disk emails you repeatedly instead of once.** When the reserve refuses an app's backup you
|
||||
are told once by the backup run — correctly — but the page showing backup status re-checks on a timer
|
||||
and sends the same message again each time. Not new: as old as the reserve itself, and seen only
|
||||
because we watched the alerts closely while proving the fix below. Harmless if the hub already
|
||||
collapses repeats — which a comment claims and nobody has checked. *(R-182)*
|
||||
|
||||
**The reserve does not guard the step that actually fills the disk.** The rule that refuses a backup
|
||||
when space runs low is checked at the wrong moment: the big write happens first, unchecked, and only
|
||||
the small write after it is refused. So the thing meant to stop a runaway backup is the thing it
|
||||
runs past. Worse, when it does refuse it says *"your last good copy is untouched"* — and we measured
|
||||
that copy being overwritten by the earlier step anyway. Found by deliberately filling a rebuilt demo
|
||||
machine. Nothing was deleted and you were emailed, both correctly. *(R-181)*
|
||||
## What shipped recently
|
||||
|
||||
**When that happens, only one page says so** — no email, no alert. The page that answers "is this app
|
||||
backed up?" is the one that stays silent. *(R-158)*
|
||||
**The backup partition is gone, and both demo machines run on the new shape.** Wiped and rebuilt on
|
||||
3 August and taken through the whole customer journey — set up, install an app, back it up, restore
|
||||
it. One storage area instead of two; the space a backup can use went from 19 GB to 65 GB on the small
|
||||
machine and 45 GB to 233 GB on the big one. Three reboots each, correct every time. The two were
|
||||
rebuilt deliberately differently — one from a local copy of the image, one by the ordinary customer
|
||||
route with the published fingerprint checked — so the disk shape and the delivery route are both
|
||||
proven, rather than one proven twice. Their previous demo apps and data are gone; that was the point
|
||||
of a wipe, and you approved it. *(R-165, R-178)*
|
||||
|
||||
**The checks now have two nets, and the second one emails you.** Every repository has one command
|
||||
that runs all of its checks; it runs by itself before every push and refuses a push that fails. That
|
||||
one lives on the workstation and can be skipped. So the build server now runs the same checks again,
|
||||
on a machine that does not care who pushed or what they typed — and **when they fail it sends you an
|
||||
email**, because a red mark on a page nobody watches is not a warning. Proven with a real broken
|
||||
change, not assumed. The one thing it still cannot do is *stop* the change: every change here goes
|
||||
straight to the main copy with no review step, so there is no point in the road for it to stand at.
|
||||
It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)*
|
||||
**What replaced the wall — and it now watches the right moment.** The wall was quietly doing a second
|
||||
job: keeping a runaway backup from eating the space the machine needs to keep running. That job is now
|
||||
explicit, and as first built it was checked too late — the big write happened first, unchecked, and
|
||||
only the small write after it was refused, while the message still promised your last good copy was
|
||||
untouched. **Fixed and proven on 3 August.** The machine now decides once, per app, **before it writes
|
||||
anything at all**, and that one answer covers all three steps: a refused app writes nothing, is not
|
||||
restarted, and the promise is now literally true — checked by fingerprinting every file before and
|
||||
after. It also stopped being blind to size, so an app is no longer waved through at 96% full and then
|
||||
allowed to write two gigabytes. Proven by deliberately filling a demo machine, once for each way it
|
||||
can refuse. Nothing is ever deleted to make room: every app has only one local copy, so "delete the
|
||||
oldest" would always mean destroying some other app's only copy. *(R-181)*
|
||||
|
||||
**A filling disk now warns the customer before anything breaks, and a failed backup now reaches you.**
|
||||
Until today the first sign that a disk was filling up was a backup that did not happen — nothing said
|
||||
anything beforehand. Two things changed. The customer is now warned while there is still room to act,
|
||||
naming the drive and how much space is left, in plain Hungarian that says what to do about it. And
|
||||
when one app's backup fails for any reason, **you** are told which app and why, with the disk figures
|
||||
attached — the page that answers "is this app backed up?" was, until now, the one page that never
|
||||
said. The customer is deliberately *not* told about that second one: they can free up space, but they
|
||||
can do nothing about a backup that failed, so telling them would only alarm them.
|
||||
**The last of the three apps that never saved their data is fixed.** Installed nowhere, so nothing was
|
||||
stranded — checked on both demo machines and in the fleet list rather than assumed. Proven by the check
|
||||
that caught it, run in both directions: it clears the fixed version and still convicts the old one.
|
||||
*(R-156)*
|
||||
|
||||
Both were proven on the demo machine by actually filling a disk. One detail is worth knowing because
|
||||
it is why there are two rules and not one: the serious warning fired when free space dropped below a
|
||||
fixed amount while the disk was only 91% full — a percentage on its own would have missed it.
|
||||
**A filling disk warns the customer before anything breaks, and a failed backup reaches you** — the
|
||||
customer while there is still room to act, naming the drive and the space left; you when one app's
|
||||
backup fails, with the disk figures. The customer is deliberately not told about the second: they can
|
||||
free space, but they can do nothing about a failed backup. Both proven by filling a real disk. There
|
||||
are two rules and not one because the serious warning fired on free space while the disk was only 91%
|
||||
full — a percentage alone would have missed it. *(R-167, R-158)*
|
||||
|
||||
**These went in *before* the partition change deliberately.** The partition being removed is also a
|
||||
barrier against a runaway backup filling the space the machine needs to run; putting the warnings in
|
||||
first means that when it comes down, the thing watching is already working and already tested.
|
||||
*(R-167, R-158)*
|
||||
|
||||
**The backup partition is gone — and both demo machines now run on the new shape.** Each was wiped
|
||||
and rebuilt from the new base image on 3 August and taken through the whole journey a customer takes:
|
||||
set the machine up, install an app, back it up, restore it. One storage area instead of two, and the
|
||||
space a backup can use went from 19 GB to 65 GB on the small machine and from 45 GB to 233 GB on the
|
||||
big one. The wall does not move; it stops existing. Each machine was rebooted three times over and
|
||||
came back correctly every time. **The two machines were rebuilt deliberately differently** — the
|
||||
first from a copy of the image held locally, the second by the ordinary route a real customer takes,
|
||||
fetching the image and checking it against the fingerprint we publish, so both the disk shape and the
|
||||
delivery route are now proven rather than one twice.
|
||||
|
||||
**Their previous demo apps and data are gone.** That was the point of a wipe and you approved it; the
|
||||
machines now carry a couple of small test apps instead.
|
||||
|
||||
**What replaced the wall.** It was quietly doing a second job — keeping a runaway backup from eating
|
||||
the space the machine needs to keep running. That job is now explicit: if a backup would push the disk
|
||||
below a safe reserve, **that one app's backup is refused, its last good copy is left exactly as it
|
||||
was, and you are told.** Nothing is ever deleted to make room; every app has only one local copy, so
|
||||
"delete the oldest" would always mean destroying some other app's only copy.
|
||||
|
||||
**How the shape was chosen — worth one line, because it was not the obvious one.** Three ways of doing
|
||||
it were built and rebooted rather than argued about. All three worked. They differed in what they
|
||||
quietly broke: one put your backups inside Docker's own storage, where the normal way of fixing a sick
|
||||
Docker would wipe them; another exposed all of Docker's internals to the part of the system that
|
||||
manages your drives. The third does neither, and costs one extra line of configuration.
|
||||
|
||||
**The new base image is now switched on**, so any machine installed from here on gets the new shape.
|
||||
The tester's box is untouched and keeps working exactly as before; it gets the new shape whenever it
|
||||
is reinstalled. *(R-165, R-178)*
|
||||
**The checks have two nets and the second emails you.** Every repository has one command that runs all
|
||||
its checks, before every push. That one can be skipped, so the build server runs them again and emails
|
||||
you on failure. It cannot *stop* a change — everything goes straight to the main copy with no review
|
||||
step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** the last app whose data was never saved; today's decisions written down.
|
||||
- **Next:** fixing the reserve so it guards the step that fills the disk, and stopping it from
|
||||
claiming your last good copy is untouched when it is not *(R-181)*. The partition merge itself is
|
||||
done and proven on both machines.
|
||||
- **After:** rebuilding how the machine records whether an app is meant to be running.
|
||||
- **Now:** both of today's items are done — the reserve and the last unsaved app. Your two decisions
|
||||
are written down and are ours to build.
|
||||
- **Next:** building those two — moving the installer onto a labelled version so publishing is one
|
||||
step you can undo, and a check that refuses to install a version nobody can download *(R-110, R-115)*.
|
||||
- **After:** the off-site copy that the machine making it can still erase *(R-95, R-87)*.
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **How a new version reaches a machine — and it has now been forgotten a third time.** Pushing the
|
||||
installer publishes it: half a minute later every new machine downloads it, with no staging and no
|
||||
way back but another push. And publishing is a step we remember rather than one the release
|
||||
performs. On 3 August the new agent — the half of the partition merge that runs on the machine —
|
||||
turned out to have been built and installed on both demo machines but **never published**, so a
|
||||
rebuild would have quietly put the *old* one back and proved a version nobody ships. Caught before
|
||||
the wipe and fixed in ten minutes, but only because someone happened to look. Nothing is installing
|
||||
today, so this is the cheapest moment to settle it. *(R-110, R-115)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **Nothing on the partition merge itself** — the shape is chosen, built, and now proven on both demo
|
||||
machines. The tester's box needs no conversion: it will simply be reinstalled. What is left is the
|
||||
reserve defect above, which is ours to fix, not yours to decide. *(R-165, R-181)*
|
||||
- **Nothing else.** You settled both open questions on 3 August — the installer moves onto a labelled
|
||||
version, and a check will refuse to install a version nobody can download. Both are written down and
|
||||
are ours to build. *(R-110, R-115)*
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-03** — Both demo machines wiped and rebuilt from the new base image, and taken through
|
||||
set-up → install an app → back it up → restore it. The backup space ceiling is gone and measured.
|
||||
The hard stop that replaced the old partition was fired for the first time on real hardware — it
|
||||
refused, it deleted nothing, it emailed you, **and it turned out to be watching the wrong step**;
|
||||
that is now the top thing to fix.
|
||||
- **2026-08-03** — The reserve now guards the step that fills the disk, and its promise is true; the
|
||||
last app whose data was never saved is fixed. Both proven on a demo machine, not just in tests.
|
||||
Earlier the same day: both demo machines wiped and rebuilt from the new base image and taken through
|
||||
set-up → install an app → back it up → restore it, with the backup space ceiling gone and measured.
|
||||
|
||||
- **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in
|
||||
a mode where reading a page cannot block a machine's status update; a one-word difference meant that
|
||||
@@ -140,11 +107,5 @@ is reinstalled. *(R-165, R-178)*
|
||||
and fixed the same day.
|
||||
|
||||
- **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were
|
||||
failing quietly. Fixed, and the arrangement that replaced them is described above.
|
||||
- **2026-08-02** — Decided: the 20 GB backup partition goes away and shares space with app data. That
|
||||
changes the disk layout, so it happens before any machine is installed outside the house. **Measured
|
||||
since:** no machine outside the house is registered yet, so this is as cheap now as it will ever be;
|
||||
and the two demo machines can simply be reinstalled rather than converted.
|
||||
- **2026-08-02** — Decided: only this machine and the tester's box are protected; every other box,
|
||||
demo boxes included, may be broken or reinstalled freely. Two of the three apps that never saved
|
||||
their data are fixed; this page created.
|
||||
failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this
|
||||
machine and the tester's box are protected, every other box may be broken or reinstalled freely.
|
||||
|
||||
@@ -86,7 +86,7 @@
|
||||
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
|
||||
| **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait |
|
||||
| **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it |
|
||||
| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.192.0** (R-165 / D-a / B2) | **IMPLEMENTED — the LAYOUT half is PROVEN-LIVE (2026-08-03); the REFUSAL half is not (R-181)** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden <local volid>`; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181**. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity |
|
||||
| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.193.1** (R-165 / D-a / B2, completed by R-181) | **PROVEN-LIVE (2026-08-03) — BOTH halves** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden <local volid>`; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181, CLOSED THE SAME DAY (controller v0.193.0 + v0.193.1) and the second half is now PROVEN-LIVE TOO.** The reserve became a **per-app, per-run ADMISSION decision** taken before the app's FIRST write and covering all three legs (DB dump, volume dump, capture) — they write under one per-app root, which is what lets one verdict cover them honestly — and it gained a **size term**, so an app is no longer admitted at 96% and then allowed to write 2 GB. **Re-proven by filling demo-hp deliberately, once for EACH term, using the method that found the defect.** *Headroom @ 08:59:46* (906 MB free / 99%): both apps refused, **the whole `backups/primary` tree byte-identical — `TREE_SHA` 111d1760c18d3440f700634ab325f8b8 before and after**, opengist's tar still at its original 182,272 B; **no `Stopping <app> for safe volume dump` line at all**, which is the positive-by-absence observable that matters because that line IS present in the 08:58 baseline run; 0 volume dumps; one alert per app, HTTP 200. Space freed, re-run @ 09:01:33 → both captured normally. *Size @ 09:03:00*, reproducing the original sequence with a real 2 GiB file in opengist's volume (previous tar **2,147,666,432 B**, the exact figure the defect was measured at) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` while **privatebin was ADMITTED and dumped normally**, proving the term is per-app rather than a global halt. **The refusal's wording was NOT weakened to fit** — the behaviour moved so the wording became true, and it is verified by tree fingerprint rather than by reading the log line, which is what lied. The `fallocate` instrument was re-proven on the rebuilt box before use (5 GiB step moved guest `df` while thin-pool `data_percent` held **36.83 → 36.83**), and teardown returned the pool to **29.43%**, below its own baseline. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity |
|
||||
| Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
|
||||
| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise |
|
||||
|
||||
|
||||
@@ -566,17 +566,40 @@ mismatch table above (`mp0` 50 G vs `mp1` 20 G) describes what a merged box no l
|
||||
before provision grew them). **Read it as a function of `mp1`, and only for a box still on the split
|
||||
layout.** Measured: `audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1.
|
||||
|
||||
**What replaced the partition's second job.** `mp1` was also a BULKHEAD: an overflow was refused per
|
||||
app with the last good unit byte-identical, and it **could not reach `/var/lib/docker`**, because that
|
||||
was a different filesystem. On a merged box it can. Decision **B2**, shipped in controller
|
||||
**v0.192.0**, is that bulkhead made deliberate — a two-term capture floor (97% used or 1 GiB free) in
|
||||
**What replaced the partition's second job — the reserve.** `mp1` was also a BULKHEAD: an overflow was
|
||||
refused per app with the last good unit byte-identical, and it **could not reach `/var/lib/docker`**,
|
||||
because that was a different filesystem. On a merged box it can. Decision **B2**, shipped in controller
|
||||
**v0.192.0**, is that bulkhead made deliberate — a two-term reserve (97% used or 1 GiB free) in
|
||||
`internal/fillwatch`'s shape, sitting beyond its critical band so the customer is always warned first.
|
||||
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
|
||||
only destroy a different app's only local copy.
|
||||
|
||||
**Status caveat, deliberately explicit:** as of 2026-08-03 **no box has been reinstalled from the
|
||||
merged golden** (R-178), so every box in the field is still on the split layout and everything above
|
||||
still describes them exactly. This subsection describes what a box built from golden ≥ 0.192.0 gets.
|
||||
**THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
|
||||
**per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
|
||||
before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
|
||||
volume dump and the recovery-unit capture**. Those three write under one per-app root
|
||||
(`backups/primary/<app>`), which is what makes one verdict able to cover them honestly.
|
||||
|
||||
- **What it guarantees.** A refused app has **nothing written for it in that run**, its previous unit
|
||||
is **byte-identical**, it is **not stopped**, nothing anywhere is deleted, and the operator gets
|
||||
**exactly one** alert naming the app, the term that bound and the disk figures.
|
||||
- **Two terms, two questions.** *Headroom*: is the filesystem already below the reserve? *Size*: would
|
||||
THIS app's write take it below? The size estimate is the app's previous `.sql` + `.tar` on disk;
|
||||
with no history the decision degrades to headroom alone, deliberately — otherwise the first backup
|
||||
is the one that can never happen.
|
||||
- **Why it is decided lazily and not once per run.** Space changes during a run: app A's dump can put
|
||||
app B under the reserve, so a verdict taken at run start reads a disk that no longer exists.
|
||||
- **Why it is never re-decided between an app's own legs.** That is precisely the shape v0.192.0 had —
|
||||
the two dump legs unguarded and only the capture refused — under which the reserve was consumed by
|
||||
the very write it exists to bound, and the refusal's *"the previous unit is untouched"* was measured
|
||||
false. Proven live on demo-hp 2026-08-03 (R-181), fixed the same day, and re-proven by filling the
|
||||
box for each of the two terms.
|
||||
- **It sits ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal
|
||||
decided inside it would already have bounced the app it is refusing to back up.
|
||||
|
||||
**Status caveat, deliberately explicit:** every box in the field that has not been reinstalled is still
|
||||
on the split layout and everything above still describes them exactly. This subsection describes what a
|
||||
box built from golden ≥ 0.192.0 gets. Both demo boxes were reinstalled from it on 2026-08-03 (R-178).
|
||||
|
||||
Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling**
|
||||
(**R-163**) — this section states the constraint, not a number. And **the same-device placement is
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -20,7 +20,7 @@
|
||||
|
||||
| ID | Item | Size | Status | Notes / map rows flipped |
|
||||
|----|------|------|--------|--------------------------|
|
||||
| R-115 | **Publishing is a remembered step — forgotten within eight hours of being documented as forgettable** | M | idea — **WAITING-ON-OPERATOR**, 2026-07-29 | A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published, so **"deployed" and "installable" are independent states that drift silently**. **Instance 1 — R-111** (morning): 17 agent releases v0.97.0–v0.113.0 stranded; a new customer would have installed without the whole R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT. Found only because the E-2d Phase 0 gate happened to look. **Instance 2 — agent 0.114.0** (same afternoon): the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, never published — which blocked Session C, since a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. **The finding is the RECURRENCE, not either instance** — both are fixed. R-111 named this leg in its own text (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; it recurred the same day, which is the evidence that **a note is not a mechanism**. **Class: → R-29, one layer up** (a control that exists and is never walked) — deliberately NOT given a second ID. **Filed as its own item rather than reopening R-111** because R-111's finding (the channel *was* stale) is closed and verified end-to-end by the E-2d install, while the process defect that caused it is a distinct problem with a distinct fix and a distinct owner. **Operator's decision, mechanisms first:** (a) publish as a step in the build/release path so deployed and installable cannot diverge; (b) a gate that refuses to deploy an unpublished+unvouched version — strongest, fails closed; (c) a session-end checklist entry; (d) accept manual + a pre-Session-C verification. **(a)/(b) are mechanisms, (c)/(d) are reminders — and R-29's whole finding is that reminders do not hold.** No code written when filed, by design |
|
||||
| R-115 | **Publishing is a remembered step — forgotten within eight hours of being documented as forgettable** | M | **READY** — operator ruling 2026-08-03: **mechanism (b), a build-side gate that REFUSES to deploy or vouch an unpublished version.** THIRD instance the same day (agent v0.120.0 deployed to both boxes while unpublished; a documented-path reinstall would have silently downgraded them and *succeeded*). CC's to build | A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published, so **"deployed" and "installable" are independent states that drift silently**. **Instance 1 — R-111** (morning): 17 agent releases v0.97.0–v0.113.0 stranded; a new customer would have installed without the whole R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT. Found only because the E-2d Phase 0 gate happened to look. **Instance 2 — agent 0.114.0** (same afternoon): the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, never published — which blocked Session C, since a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. **The finding is the RECURRENCE, not either instance** — both are fixed. R-111 named this leg in its own text (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; it recurred the same day, which is the evidence that **a note is not a mechanism**. **Class: → R-29, one layer up** (a control that exists and is never walked) — deliberately NOT given a second ID. **Filed as its own item rather than reopening R-111** because R-111's finding (the channel *was* stale) is closed and verified end-to-end by the E-2d install, while the process defect that caused it is a distinct problem with a distinct fix and a distinct owner. **Operator's decision, mechanisms first:** (a) publish as a step in the build/release path so deployed and installable cannot diverge; (b) a gate that refuses to deploy an unpublished+unvouched version — strongest, fails closed; (c) a session-end checklist entry; (d) accept manual + a pre-Session-C verification. **(a)/(b) are mechanisms, (c)/(d) are reminders — and R-29's whole finding is that reminders do not hold.** No code written when filed, by design |
|
||||
| R-116 | **The drive-absent alarm and its recovery are a mismatched pair — generic on the way out, specific on the way back** | S | idea — **PROVEN LIVE 2026-07-29** | Absent fires `storage_disconnected`; return fires `backup_target_restored`. `backup_target_absent` never fires at all (count 0 across a full Session-C run), so an operator gets an alarm they cannot match to its recovery — exactly what `notifyDriveReturned`'s own comment forbids. Root cause: `notifyDriveAbsent` (`intermediary.go:635-646`) branches on `isTarget[a.Path]` with `a.Path` the GUEST path, and `driveTargetByPath` (`:602-616`) builds it as `out[GuestPath] = d.BackupTarget` — but **the drive is TWO `/disks` rows and the flag and the guest path sit on different ones**: the `felhom-backup` storage row has `BackupTarget: true` (`felhom-agent/internal/localapi/disks.go:211`) and gets a guest path only while classified user-data, while the registry union row has the guest path and **never assigns `BackupTarget`** (`disks.go:265-267`). Absent ⇒ the flagged row loses its guest path ⇒ the union row writes `false` ⇒ generic. On return the rows rejoin ⇒ specific. v0.184.1 fixed the KEYING, not this. **Only reachable because R-113 made the gate fire at all.** Fix likely agent-side; decide the repo first. Blocks E-2's C5. Evidence: `audits/SESSION-C-2026-07-29.md` §5 |
|
||||
| R-113 | **The drive-absent gate cannot fire on device loss — E-2b's alarm is wired to an unreachable condition** | M | idea — **PROVEN LIVE 2026-07-29** | `planDriveGates` (`felhom-controller/internal/web/intermediary.go:216-262`) treats a path as present by OR-ing in `d.BoundUnderParent`, which the agent derives from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`internal/localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is not device-bound and its mountinfo entry outlives the device**, so the gate reads it as present and `notifyDriveAbsent` is never called. Live on a fresh box: target drive hot-detached, agent said `enrolled drive absent by UUID` every 20 s for 4½ min, controller logged **0** `[gate]` lines, hub received **zero** events — neither `backup_target_absent` nor the generic `storage_disconnected`. Not a virtualisation artefact (device-bound-mount vs manual-bind is the same on metal); caveat: SCSI hot-detach, physical unplug not staged. **Sixth instance of seam-built-but-never-wired — E-2b wired the seam to a condition that cannot occur.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 |
|
||||
| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
|
||||
@@ -157,18 +157,18 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
|
||||
| R-97 | **The whole-guest backup tier has NO failure signal to the hub — `internal/quiesce` never notifies** | S | **SHIPPED (controller v0.177.0 + hub v0.78.0, 2026-07-27)** — **R-97a:** `quiesce.TierNotifier`, a seam (not an import) wired by an init-only setter, edge-triggered on the R-88 breaker ARMING so a failing tier is reported once per run rather than once per retry; recovery rides `recordSuccess`'s existing bool. **NEW operator-only event types** `whole_guest_backup_failed`/`_recovered` — deliberately NOT `backup_failed`, which carries a customer Hungarian template AND sits in demo-felhom's live `enabled_events`, so reusing it would have emailed the CUSTOMER about a backup they cannot act on while it was still retrying. The recovery joins `recoveredPairedDownTypes` because its `info` severity would otherwise be dropped by `severityNotifies` — the operator would hear it break and never hear it heal. **The hub's operator cooldown was keyed `customerID:eventType` alone**, so one tier would have masked the other for an hour; now narrowly extended with a `tier` suffix taken from the event details, leaving every other event type unchanged. **R-97b:** a suppression window keyed to the quiesce CYCLE (not a state test — v0.164.0's `!= StateStopped` filter cannot see an app caught MID-RESTART, which is exactly how BookStack alarmed), consumed at the same single derivation point `classifyRunStates`. Grace = **180 s**, derived from the deploy flow's 120 s health timeout and Mealie's 60 s `start_period`; it **expires**, so an app that genuinely fails to come back still alarms. **PROVEN LIVE end-to-end with a control:** the new type POSTs 200 from inside guest 9201 while a bogus type 400s, and `notification_log` shows **1 operator row, 0 customer rows**. The quiesce→notify link itself is unit-proven only. | On 2026-07-27 three whole-guest backups failed and three quiesce cycles stopped and restarted every customer app stack, and **not one `backup_failed` event reached the hub.** It is not the allowlist — the hub already carries `backup_failed` and `backup_completed` (they are emitted by the controller's *app-data* backup path). The cause is that **`internal/quiesce` does not import `internal/notify` at all**: the tier R-82 built has no route to the hub, so a whole-guest backup can fail indefinitely in silence. The loop's only trace was `app_start_failed` — **info** severity, **Hungarian**, on the **customer** channel — telling the customer BookStack was down (it had been caught mid-restart by the third cycle) without saying why, during an outage the system itself caused. So the one signal that did fire was both the wrong tier and the wrong story. **Shape:** emit `backup_failed`/`backup_completed` from `quiesceAndPollTiers` naming the TIER, operator-tier; and decide whether a quiesce-induced restart should suppress `app_start_failed` the way controller **v0.164.0**'s deliberate-stop filter does — an app the backup stopped on purpose is not a fault. R-88's breaker bounds the repetition but changes nothing about the silence |
|
||||
| R-95 | **The restic offsite tier's credential CAN DELETE — R-89's "parallel question", now ANSWERED** | M | idea — established read-only 2026-07-27 | **The exposure closed on the weekly PBS tier is fully open on the daily restic tier**, which holds the customer's actual documents and photos and is the only tier that survives losing the box. Established without mutating anything: **(1) Identity** — a per-customer *subaccount* on `storage-box-pool-1` (box 611714, bx11, `u629488`): `u629488-sub1` home `felhom-demo-felhom`, `sub2` peti-felhom, `sub3` demo-hp, each labelled `felhom-customer`. Auth is an **SSH key stored ON THE BOX** (`…/felhom-controller-data/_data/data/offbox/ssh_key`, 0600, beside `repo_password` + a pinned `known_hosts`) — customer-side, not hub-side, so a compromised guest holds it. **(2) Read-write: YES** — the API reports **`readonly=False` on all three subaccounts**, and it is not merely latent: the controller runs `restic forget --group-by host,tags --keep-daily 7 --keep-weekly … --prune` **from the box** (`backup/offbox.go:984`, also `:1070`). Delete rights are exercised on every run. **(3) Append-only: NO, and not expressible** — the repo is built as `sftp:` (`offbox.go:482`); restic's append-only mode requires the **REST server** backend, which plain SFTP cannot provide. **(4) A zero-code mitigation exists and is unused:** the box type carries `snapshot_limit=10` and the API reports `snapshot_plan=null` with **0 snapshots** and `size_snapshots=0`. Hetzner Storage Box snapshots are taken **server-side, outside the SFTP namespace** — an SFTP subaccount cannot delete them — so they are a genuine immutability layer at no extra cost and with no code change. **Rule once for both tiers, per R-89.** Options, cheapest first: enable a snapshot plan (operator click, immediate); split backup-write from prune so pruning runs somewhere the box cannot reach; or move the repo to restic's REST server with `--append-only`. Flips the capability-map row for offsite immutability |
|
||||
| R-94 | ~~A hand-synced version constant drifts, and the gate that would catch it is never run~~ | XS | **CLOSED — SHIPPED hub v0.87.0, 2026-08-02** | Closed by **deleting** the label rather than deriving it: the Setup command fetches the installer at run time from a 30 s-git-synced website (R-110), so no build-time value in the hub can be true. `hostinstall_gates.py` gate 1 inverted to pin the ABSENCE of a version literal; the tautological `render_test.go` assertion deleted (demonstrated passing at `9.9.9`). Detail: `OPEN-ITEMS.md` R-94 |
|
||||
| R-110 | **`main` is the installer's publish channel — there is no staging** | S | idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)** | `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a `--period=30s`, and nginx serves that working tree directly (`location /scripts/`, `root /usr/share/nginx/html/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`, `:1262`; `runbooks/day0-install.md` C.1) receives. **There is no tag, no pinned-version path, no staging copy and no rollback other than another push** — for the one artifact that runs as **root on a virgin box**, the most privileged thing Felhom ships. **Two consequences worth stating plainly:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29, so the proof run confirms what customers already receive rather than clearing it for release; and the precaution the old R-94 row recorded ("do not point every new box at an installer that has never run") **was never available to take**, because nothing points boxes at a version. **Open question for the operator, not a defect to fix blind:** should `/scripts/` serve a pinned release — a tag-tracked git-sync ref, or a versioned directory (`/scripts/1.22.0/…`) with the hub's generated command naming a version — or is `main`-tracking the accepted shape for a one-operator product where the alternative is a release ritual nobody performs? **Exposure today is zero** (no boxes are installing), which is exactly why it is cheap to decide now. Whichever way it goes, it decides whether R-94 leg (a) makes the label a *fact* (derived from the served script) or keeps it a *claim*. Flips no capability-map row — the map states what the platform does, and this changes nothing about that |
|
||||
| R-110 | **`main` is the installer's publish channel — there is no staging** | S | **READY** — operator ruling 2026-08-03: **option (b), the channel moves to a TAG**, so publishing is moving the tag and rollback is moving it back. **Must cover BOTH channels** — the nginx `/scripts/` git-sync AND the nine files the installer fetches from `raw/branch/main` — or it only half-works. CC's to build | `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a `--period=30s`, and nginx serves that working tree directly (`location /scripts/`, `root /usr/share/nginx/html/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`, `:1262`; `runbooks/day0-install.md` C.1) receives. **There is no tag, no pinned-version path, no staging copy and no rollback other than another push** — for the one artifact that runs as **root on a virgin box**, the most privileged thing Felhom ships. **Two consequences worth stating plainly:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29, so the proof run confirms what customers already receive rather than clearing it for release; and the precaution the old R-94 row recorded ("do not point every new box at an installer that has never run") **was never available to take**, because nothing points boxes at a version. **Open question for the operator, not a defect to fix blind:** should `/scripts/` serve a pinned release — a tag-tracked git-sync ref, or a versioned directory (`/scripts/1.22.0/…`) with the hub's generated command naming a version — or is `main`-tracking the accepted shape for a one-operator product where the alternative is a release ritual nobody performs? **Exposure today is zero** (no boxes are installing), which is exactly why it is cheap to decide now. Whichever way it goes, it decides whether R-94 leg (a) makes the label a *fact* (derived from the served script) or keeps it a *claim*. Flips no capability-map row — the map states what the platform does, and this changes nothing about that |
|
||||
| R-128 | **`ISO_VERSION` "aligns with SCRIPT_VERSION" was a comment nothing evaluated** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | Closed by **correcting the claim, not asserting it**: the ISO is frozen while `felhom-host-install.sh` is fetched at run time from `main` (R-94/R-110), so an assertion would invent a constraint. `build-felhom-iso.sh:45-52`. Full reasoning in `OPEN-ITEMS.md` |
|
||||
| R-154 | **`[first-boot]` is automated-install-only, and nothing in the tree said so** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | A PVE property, measured with a same-image control (`audits/SPIKE-universal-iso-3-2026-07-31.md` §2); recorded at `scripts/iso/pkg/build-deb.sh:6-11`. Superseded in practice by the `.deb` delivery route |
|
||||
| R-155 | **`iso-repack.sh` refused any ISO without `auto-installer-mode.toml`** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | **Narrowed, not deleted** — unchanged for `FELHOM_MENU=single` (`iso-repack.sh:121-128`), does not apply to `release` where the file's absence *is* gate G1. Do not remove it wholesale |
|
||||
| R-156 | **An app's data is neither persisted nor backed up, and it reports healthy** — a template mounts a volume at a path the app never writes, so data sits in the container's writable layer: lost on redeploy, tarred nightly as an empty dir, healthcheck green | S | **DETECTED + GATE SHIPPED; papra REFERRED** (catalog sweep, 2026-08-02) | Found by Campaign 10 on **papra**; the 53-template sweep convicted **gramps-web** and **wishlist** too and **fixed both**. The gate is `app-catalog-felhom.eu/scripts/check-volume-persistence.py` — a runtime probe (`docker diff` + mount occupancy + writability, canary self-test, fails closed); the detector and the gate are one program. **papra is NOT fixed**: the fix needs the app to use `/app/data` or the template to mount `/app/app-data`, so it is referred. Sweep evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` (53 probe.json). Ranking in `OPEN-ITEMS.md` |
|
||||
| R-156 | ~~**An app's data is neither persisted nor backed up, and it reports healthy**~~ | S | **CLOSED — all three apps fixed, 2026-08-03.** papra's template now mounts `papra_data:/app/app-data` (the app's own data ROOT, chosen over reconfiguring three env vars so a future upstream path cannot escape). Deployed nowhere — re-checked on both demo guests and the hub fleet view, not inherited. Gate run in BOTH directions: CLEAN fixed, BROKEN reverted | Found by Campaign 10 on **papra**; the 53-template sweep convicted **gramps-web** and **wishlist** too and **fixed both**. The gate is `app-catalog-felhom.eu/scripts/check-volume-persistence.py` — a runtime probe (`docker diff` + mount occupancy + writability, canary self-test, fails closed); the detector and the gate are one program. **papra is NOT fixed**: the fix needs the app to use `/app/data` or the template to mount `/app/app-data`, so it is referred. Sweep evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` (53 probe.json). Ranking in `OPEN-ITEMS.md` |
|
||||
| R-157 | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover** | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.189.0 + v0.190.0, 2026-08-02)** | Both mechanisms closed. **B:** the container count → recorded intent (R-166). **A:** the single T+5 s observation → a settle-then-sweep window (sample every 5 s, settled after 3 identical samples, one sweep at the end, ends on settled OR a 50 s budget). Budget is 50 s because settle+budget+one retry must fit the 90 s `deadAppBootGrace`; a test rejected 60 s at 95 s. Late recoveries are REPORTED, not hidden by widening the grace. **The fix had its own defect, found live:** sampling the Manager's 10 s-refreshed cache let "settled" mean "the cache did not update" — `sampleBootFleet` now refreshes first. **6/6 hard resets clean on the shipped build**, plus a same-app before/after (missed 18:08:35, recovered 18:18:50) |
|
||||
| R-170 | **The drive-backed boot gate infers a Stop from a container count** | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | `shouldRecreateOnBoot` reads `desired_state` with the same three-way table as `isBootOrphan`; absent keeps the old `hasContainers` behaviour exactly; `presentStable` untouched. Agreement pinned from both sides against one fixture table. Live: calibre-web (`running`, zero containers) recreated and immich (`stopped`) left alone in the same reboot |
|
||||
| R-171 | **The boot sweep started apps whose data drive was ABSENT** — a regression from v0.189.0 | S | **CLOSED — SHIPPED + PROVEN-LIVE (v0.190.0, 2026-08-02)** | Reasoned from the diff, CONFIRMED on hardware first (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The write hazard was blocked only by an ACCIDENTAL filesystem permission (host-root-owned mountpoint + unprivileged guest); the false dead-app alarm was real on every box. New fail-safe `bootrecon.StartGate` seam, also covering quiesce and in-flight app-data operations (§8.2). The rule already existed on the API path (`startGatedByMissingDrive`); the sweep bypassed it |
|
||||
| R-166 | **App state gets a desired/observed model with its own store** (operator decision D-b) | M | **SHIPPED + PROVEN-LIVE — controller v0.189.0, 2026-08-02** | Tri-state `desired_state` in `app.yaml`, written ONLY by the customer's action; `bootrecon` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN** so legacy boxes keep byte-identical behaviour; running-only backfill; `backup.AppStopGuard` covers every stop→work→start window in its own marker file. **The durable fix for R-157 mechanism B.** Both blocking facts answered at source first: the crash-safe journal pattern DID already exist (quiesce, migrate) but covered **none** of the app-data path, and the SQLite store was rejected as a home because `metrics.db` is optional by design. `07`/`02` architecture docs carry the desired/in-flight/observed split. **Left for R-170:** the drive-backed boot gate still uses the container-count inference |
|
||||
| R-158 | ~~A local Tier-1 backup failure reaches no hub channel~~ | S | **SHIPPED — controller v0.191.0 + hub v0.89.0, 2026-08-02** | Collapsed per the lifecycle rule. Closed by **R-167** (D-c's operator half) — no second row was filed for the same wire. New `unitNotify` seam fired per app from `captureAllRecoveryUnits` with the loop continuing, carrying the target filesystem's used/free bytes. **Routed to a new OPERATOR-ONLY type `recovery_unit_capture_failed`, NOT the `backup_failed` this row proposed** — that type is customer-enabled by default and would email the customer about a failure they cannot act on; decision D-c overrides the proposal. **Flips:** the new capability-map row *"A failed per-app Tier-1 backup reaches the OPERATOR"* → **PROVEN-LIVE** (`customer | … | skipped | operator_only` observed in the hub's `notification_log`, 9201, 2026-08-02) |
|
||||
| R-167 | ~~Storage monitoring and backup alerts (decision D-c)~~ | M | **SHIPPED — controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02** | Collapsed per the lifecycle rule. **Landed BEFORE the R-165 merge rather than with it** — D-a's condition (2) requires the same step and never after, and first is strictly better: the warnings were proven on hardware while the wall is still standing. Customer half = `internal/fillwatch`, per filesystem, two threshold terms (85% / 5 GiB; critical 95% / 2 GiB), edge-triggered with persisted state and a 75% / 7 GiB hysteresis return. **It emits the pre-existing `disk_warning`/`disk_critical` pair, which had NO producer in any repo — the sixth *built-but-never-wired* instance**; the hub's two generic `customerMessages` entries were deleted so the dynamic Hungarian survives. Operator half = R-158. **Flips:** two new capability-map rows → **PROVEN-LIVE**. **Follow-ups:** R-177 (no run-now path), R-175 (§7.5's bound is one box's) |
|
||||
| R-165 | Merge `mp1` into `mp0` — the dedicated backup partition stops existing (decision D-a) | M | **SPIKED 2026-08-02 — WAITING-ON-OPERATOR** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md`, M1-M5, **no layout touched**. Prerequisite R-167 is now SHIPPED. **Three findings the merge session must not re-derive:** (1) *"the layout" is not one thing* — demo-felhom `200G/50G`, demo-hp `50G/20G`, golden `16G/8G`, so any fixed pair is already wrong for one box (→ R-175); (2) **`mp1` is a BULKHEAD, not only a ceiling** — an overflow today cannot reach `/var/lib/docker`, and after the merge it can, which is the one place "the merge is cheap" stops being true; (3) the golden fails closed on the split in **four** places, not one. **M5: D-a's condition (1) is currently SATISFIED** — no external box is in the hub's host register, and both demo boxes are Tier 0 and reinstallable, so migration cost for the measurable population is zero. **Recommendation: S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), fresh-install shape.** **Two prerequisites unmeasured → R-176.** Does NOT close R-163 until it lands |
|
||||
| R-165 | ~~Merge `mp1` into `mp0` — the dedicated backup partition stops existing (decision D-a)~~ | M | **CLOSED — PROVEN-LIVE 2026-08-03.** Variant V-c shipped (golden v3.0.0 + agent v0.120.0 + controller v0.192.0); both demo boxes reinstalled and proven end to end (R-178). **Its stated replacement for the bulkhead — B2 — was completed by R-181 the same day** (controller v0.193.0/.1): the reserve became a per-app, per-run admission covering all three write legs and gained a size term, and both terms were re-proven by filling demo-hp. R-181 is collapsed into this row rather than carried separately | `audits/SPIKE-r165-mp1-merge-2026-08-02.md`, M1-M5, **no layout touched**. Prerequisite R-167 is now SHIPPED. **Three findings the merge session must not re-derive:** (1) *"the layout" is not one thing* — demo-felhom `200G/50G`, demo-hp `50G/20G`, golden `16G/8G`, so any fixed pair is already wrong for one box (→ R-175); (2) **`mp1` is a BULKHEAD, not only a ceiling** — an overflow today cannot reach `/var/lib/docker`, and after the merge it can, which is the one place "the merge is cheap" stops being true; (3) the golden fails closed on the split in **four** places, not one. **M5: D-a's condition (1) is currently SATISFIED** — no external box is in the hub's host register, and both demo boxes are Tier 0 and reinstallable, so migration cost for the measurable population is zero. **Recommendation: S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), fresh-install shape.** **Two prerequisites unmeasured → R-176.** Does NOT close R-163 until it lands |
|
||||
| R-159 | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy** | XS | **SHIPPED** (`templates/wishlist/`, 2026-08-02) — filed for the CLASS | Image declares `VOLUME /usr/src/app/data`; template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns names only for compose-declared volumes, so `DumpAppVolumes` never sees an anonymous one. **The class is open:** any image `VOLUME` at an unmounted path is silent unbacked-up storage — **`immich-server` has one today** at `/data`, empty when measured. Proposed `REUSE.md` rule: a template mounts every path in `Config.Volumes`, or says why not |
|
||||
| R-160 | **gramps-web persisted three paths and wrote to none of them** | XS | **SHIPPED** (`templates/gramps-web/`, 2026-08-02) | `/app/data` appears nowhere in the image's env; the accounts DB and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths, the template three, one a phantom. **Severity above papra's:** papra loses documents the customer may hold elsewhere; gramps-web loses the artefact built inside the app, of which no other copy exists by construction |
|
||||
| R-161 | **The volume-persistence gate is enforced by convention, not automatically** | M | **RULED + SHIPPED at reduced scope** (operator, 2026-08-02; `app-catalog` `fd7747d`) | Enforcement is **convention**: the catalog repo has no CI at all (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none). **R-29's record: three orphaned gates, one enforced, and only the enforced one ever stopped anything — so the ruling copied the shape that works.** **Controller-side REJECTED with a measured reason:** a check at template load can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — it would pass on the very defect it exists to catch (the property is runtime-only; see `check-volume-persistence.py`'s header). **CI REJECTED for now** — neither repo has any, no users yet. **Shipped:** `scripts/catalog_gates.py`, one entry point over all three gates, non-zero on any failure, mandated in the catalog's `CLAUDE.md` as `site_gates.py` is. **Open residue is only the automatic half**, sufficient while one person touches templates |
|
||||
|
||||
Reference in New Issue
Block a user