admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: R-181 closed, R-156 closed, R-110 + R-115 rulings recorded, R-182 filed
gates / gates (push) Successful in 7s
Details

Browse Source 
R-181 CLOSED (controller v0.193.0 + v0.193.1) and proven live on demo-hp for
BOTH reserve terms. The reserve is now a per-app, per-run ADMISSION decision
taken before the app's first write and covering all three write legs, and it
gained a size term. The refusal's wording was not weakened; the behaviour moved
so it became true, verified by sha256 tree fingerprint.

R-156 CLOSED — papra's template mounts the app's own data root. Precondition
re-measured rather than inherited (both boxes were wiped today).

Part 4, documentation only, nothing built:
- R-110 WAITING-ON-OPERATOR -> READY. Ruling: option (b), the installer's
  publish channel moves to a TAG. Recorded with the condition that decides
  whether it works at all — it must cover BOTH the /scripts/ git-sync and the
  nine files the installer fetches from raw/branch/main.
- R-115 WAITING-ON-OPERATOR -> READY. Ruling: mechanism (b), a build-side gate
  refusing to deploy or vouch an unpublished version. The third instance (agent
  v0.120.0) would have silently downgraded both demo boxes while succeeding.

R-182 NEW: the periodic status refresh has no admission scope, so a refused app
re-alerts on every poll (measured: a second alert pair 13s after the run's).
Pre-existing in v0.192.0; deliberately not fixed in the R-181 task.

capability map: the local-backup row moves to PROVEN-LIVE in BOTH halves.
ROADMAP: R-165 collapses to CLOSED; R-181 collapsed into it.
07-backup-architecture.md: the reserve's contract stated as what the code
provides (S-1 — an architectural contract changed in the same session).
STATUS.md trimmed 150 -> 111 lines, "What's broken" no longer holds shipped
work, and the stale "After:" line (pointing at work that shipped on 2 August)
is fixed.
This commit is contained in:
authentik Default Admin
2026-08-03 11:36:16 +02:00
parent 06cbf8df29
commit fb652024ea
7 changed files with 324 additions and 449 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+195 -337
View File
@@ -1,358 +1,216 @@
# REPORT — R-178: both demo boxes reinstalled from the merged golden and proven (2026-08-03)
# REPORT — R-181 (the reserve guards the write that fills the disk) + R-156 (papra) + two operator rulings
**Overwritten** per the standing rule. The prior contents (R-165, the bake, 2026-08-03) have their
durable record in `documentation/backlog/OPEN-ITEMS.md` R-165 and the per-repo CHANGELOGs.
**Date:** 2026-08-03 · **Repos:** `felhom-controller` (v0.192.0 → **v0.193.1**), `app-catalog-felhom.eu`, `felhom.eu` (docs only — **no hub change, no hub version bump**)
**Runbook, not a task.** No repo got a version bump and nothing was built. One artifact was
**published** (agent 0.120.0) on an explicit operator ruling — see §2. All code findings are filed as
register rows, not commits, per the runbook's §7.
## 1. Baselines — re-read on arrival, all matched §1
---
## 1. Preconditions P1–P6, each as measured
| # | Precondition | Measurement |
|---|---|---|
| **P1** | Golden sha256 matches R-178's record | **PASS** — computed from the artifact itself on demo-hp: `sha256sum` → `54e2a4c431daf580d2807b82d810be36a7c6a9f697b27094dc2912d26a43b3e0`, 649,547,835 B. Matches R-178's recorded prefix and the hub manifest's stored value exactly. Re-verified after copying to `local`: identical |
| **P2** | A rollback golden still exists | **PASS, three depths.** Split-layout golden **0.188.0** fetchable from Gitea (`HTTP 200`, 649,310,288 B). A **local** split-layout golden sat on each box — `local:backup/vzdump-lxc-9100-2026_07_21-18_24_49.tar.zst` (demo-hp), `…2026_07_20-17_50_57…` (demo-felhom). Best: **full guest vzdumps**, three per box, newest `vzdump-lxc-9201-2026_08_03-04_54_28.tar.zst` (2,351,557,870 B, demo-hp) and `…2026_08_03-04_44_50…` (6,323,506,165 B, demo-felhom), both ~3 h old — so each box could be put back exactly as it was |
| **P3** | Neither demo box holds anything wanted | **PASS, stated explicitly.** demo-hp: paperless-ngx (webserver/postgres/redis) + filebrowser + samba shares. demo-felhom: immich (×4), docmost (×3), calibre-web, bookstack (×2) + filebrowser + samba. Both are demo customers (`Demo HP`, `Demo Ügyfél`); no real customer data. **All of it was destroyed by the wipes and none of it was restored** — that was the point, and the operator confirmed each wipe |
| **P4** | The colleague's box untouched | **PASS** — `peti-felhom` is Tier 2 and out of scope. No command was sent to it; it appears in the hub customer list as `DOWN`, exactly as before |
| **P5** | Both hosts' agent is v0.120.0 | **PASS** — `felhom-agent 0.120.0` on both, before and after |
| **P6** | The hub's host register, BEFORE | **Captured**: `demo-felhom-8363b5 / Demo Ügyfél / 0.120.0 / ONLINE / 1-of-2 guests`; `demo-hp-bb76ea / Demo HP / 0.120.0 / ONLINE / 1-of-2`; `drill-r50-0a4f9a / drill-r50 / 0.113.0 / DOWN` (the fixture, untouched throughout) |
---
## 2. Two findings that contradicted the runbook's premise, both surfaced before any wipe
**(a) The golden was ALREADY vouched.** Hub log, a positive observable:
`2026/08/03 07:23:26 [INFO] Artifact manifest set: agent=0.119.0 golden=0.192.0 min_agent="0.113.0"
wrapper_sha=true` — roughly ten minutes before this session's first hub read, and this session had
POSTed nothing. The configuration page confirmed `Artifacts.GoldenVersion=0.192.0`,
`GoldenSHA256=54e2a4c4…3b3e0`. So §7's prove-then-vouch order was already spent, and Phase A's stated
safety ("nothing is official yet, so a failure reaches nobody") was void. **Operator ruling: accept it
and proceed**, keeping the two different supply paths. Recorded on `CONTEXT.md` S-14, now marked SPENT
with the reason — the rule lived only in prose and nothing enforced it.
**(b) Agent v0.120.0 had never been published — the vouched agent was 0.119.0.**
`GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200). Installer step 5's
idempotent skip requires `installed == vouched` exactly, so a documented-path reinstall would have
**downgraded both boxes** to the pre-merge 0.119.0 — and would have *succeeded* while doing it, since
the current `step_grows` sets `SYSDATA_GROW=0` and 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on
error) therefore never fires. The session would have proven a stack nobody ships. **Operator ruling:
publish and vouch first.** `scripts/publish-agent.sh 0.120.0` from a clean tree at `4bb84fc3`
(`git status --porcelain` empty, `HEAD == origin/main`): upload HTTP 201, **round-trip GET verified**,
`AGENT_SHA256=a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`. Vouched via
`POST /configuration/artifacts` → `303 …flash=artifacts_set`, hub log
`Artifact manifest set: agent=0.120.0 golden=0.192.0`; the hub resolved the sha authoritatively from
Gitea rather than trusting the submitted value. Filed as the **third instance of R-115**, not a new
ID — R-115 is that row, and it has been WAITING-ON-OPERATOR since 2026-07-29.
---
## 3. demo-hp — the layout proof
**Install path.** Uninstall: `./felhom-host-install.sh --uninstall --vmid 9201` (typed-vmid
confirmation supplied over a pty). Install:
```
./felhom-host-install.sh --customer-id demo-hp --mode appliance --vmid 9201 \
--cores 7 --memory 26906 \
--golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst \
--passphrase-file /root/.pp-demo-hp
```
Script fetched from `https://felhom.eu/scripts/felhom-host-install.sh`, **v1.22.0**, sha256
`ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4` — byte-identical to the repo copy
that was read. The passphrase went file→file into a 0600 file and never onto a command line. A
`--dry-run` preceded the real run and resolved `manifest: agent v0.120.0 (sha a7763d31…), golden
v0.192.0` with `grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)`.
**One deviation, mine, and it cost a restart.** The first attempt staged the golden on
`felhom-backup` with `--archive-storage felhom-backup`. Pre-flight passed; **step 8/8** failed:
`HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)` —
the default pool-scoped ACL grants `local local-lvm felhom-pbs` only. Fixed by copying the golden to
`local` (sha re-verified after the copy) and `--resume`. Filed as **R-180**: the condition is
statically checkable in pre-flight, and the failure lands *after* step 4b has rotated and vaulted
root@pam.
**Layout evidence.**
```
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G # and NO mp1 line
rootfs: local-lvm:vm-9201-disk-0,size=32G
/var/lib/felhom /dev/mapper/pve-vm--9201--disk--1 ext4 rw,relatime,stripe=16
/var/lib/docker /dev/mapper/pve-vm--9201--disk--1[/docker] ext4
/mnt/sys_drive /dev/mapper/pve-vm--9201--disk--1[/sys_drive] ext4
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/felhom
/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/docker
/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /mnt/sys_drive
stat -c %d → 64519 for all three paths
/etc/fstab: /var/lib/felhom/docker → /var/lib/docker ; /var/lib/felhom/sys_drive → /mnt/sys_drive
both binds writable (touch succeeded on each)
```
`/mnt/sys_drive` shows a second `findmnt` row — it is the controller container's
`-v /mnt:/mnt:rslave` propagation (peer group 383 vs the fstab bind's 333), the same shape the split
layout had, not a stacked bind.
**Reboots — each individually, minimum three:**
| # | started | controller healthy | outcome |
| Repo | `main` @ arrival | Version | Shipped |
|---|---|---|---|
| 1 | 08:13:33 | 08:13:55, `Up 12 seconds (healthy)` | one filesystem, 69G/65G on both paths |
| 2 | 08:13:59 | 08:14:15, `Up 7 seconds (healthy)` | same |
| 3 | 08:14:19 | 08:14:35, `Up 7 seconds (healthy)` | same |
| `felhom-controller` | `4be6467b501b` | v0.192.0 | **v0.193.0 `fef07c3`** → **v0.193.1 `6c43bf6`** |
| `app-catalog-felhom.eu` | `7cb58ecdf8e7` | n/a | `122bbee` |
| `felhom.eu` | `6b5d64c1fa73` | hub v0.89.0 | docs only, **no bump** |
After all three, `mountpoint -q` returns true for **all three paths** and both binds still resolve to
the single volume's subdirectories — which is what the reboots exist to test. `uptime -s` =
`2026-08-03 06:14:24 UTC`, matching reboot 3, so these were real reboots.
All three clean (`git status --porcelain` empty, `HEAD == origin/main`) before every build.
**The journey — method stated: endpoint-level, not a browser.** `claude-in-chrome` does not exist on
DooPlex; every step below invoked the exact endpoint the dashboard's own JavaScript calls.
## 2. The fix
| Leg | Endpoint | Observable |
|---|---|---|
| Claim | `POST /claim` with the pre-auth HMAC CSRF token (64 chars) **and** its `felhom_claim_csrf` cookie | `302 → /`; gate discriminator flipped `{"error":"dashboard not yet claimed"}` → `{"error":"authentication required"}`. **The code is emailed-only (R-119) — the operator supplied it**, after an operator resend rotated the generation (the previous code had been consumed 12 d earlier) |
| Deploy | `POST /api/stacks/<n>/deploy` with `{"values":{…}}`, field values scraped from the deploy form exactly as the form's own `fetch` does | `{"ok":true}`; `privatebin Up (healthy)`, then `opengist Up (healthy)` |
| Back up | `POST /api/debug/backup/dbdump` — this runs the **production** `RunDBDumps` path (DB leg → `runVolumeDumps` → `captureAllRecoveryUnits`); the debug route only starts it instead of waiting for 03:30 | `Volume dump: opengist/… → 178.0 KB`, `privatebin/… → 2.5 KB`, `App-data backup completed … 2 volume dump(s) (3.962s)`, `Recovery unit captured for …` ×2 |
| **Restore** | `POST /backup/restore` `stack_name=privatebin snapshot_id=primary` | A marker planted in the live volume was **deleted**, then restored: `{"ok":true,"message":"privatebin visszaállítva (primary)."}` in **9.2 s**, and the marker returned with an **identical sha256 `ac1faae6134871d640d5cb1bbc6b5092d1e96b7e46c6c3bcb6e8236cc33ae861`**. App `Up (healthy)` afterwards |
**One admission verdict per app per run** (`controller/internal/backup/admission.go`), taken before
that app's **first** write and consulted by all three legs — DB dump, volume dump, unit capture. The
three write under one per-app root (`appbackup.RecoveryUnitPath`), which is what makes one verdict
able to cover them honestly.
**Recovery unit path, and it lands on the single volume:**
`/mnt/sys_drive/felhom-data/backups/primary/<app>/{manifest.json,compose/,volume-dumps/}`, whose `df`
is `/dev/mapper/pve-vm--9201--disk--1` — the merged volume.
- **Lazy, not run-wide.** App A's dump can put app B under the reserve; a run-start verdict reads a
disk that no longer exists. **Never re-decided between an app's own legs** — that is the split being
closed. **Reset per run.**
- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act, so a refused app is never
bounced. **After** the volume-less check, which has no write to gate.
- **Exactly one operator alert per refused app per run.** Leg order unchanged.
- **Size term added:** *would this app's write cross the reserve?* — estimated from its previous
`.sql` + `.tar`. **No history → headroom-only**, or the first backup becomes the one that can never
happen; the alert says so when that applies.
**Note on the first app chosen.** `privatebin`'s catalog entry declares no `backup:` section, so its
first per-stack capture produced `"volume_dumps": null` — correct for that declaration, not a defect,
but it means a per-stack `POST /stacks/<n>/backup` writes compose+config only; the volume leg lives in
the full pass. A second app (`opengist`) was deployed so the run had both a capture and, later, a
refusal.
## 3. Files
**The ceiling is gone, measured:** a recovery unit can use **65 GiB** — the whole volume — against the
**19 GiB** the pre-wipe `mp1` slice offered (`/dev/mapper/pve-vm--9201--disk--2 20G 95M 19G 1%`).
---
## 4. The floor's first live firing — and it does not do what it says
**Instrument, proven before use.** demo-hp's thin pool is **53.93 GiB** and the auto-sized volume is
70 G, so a real fill to 97 % would have exhausted the pool and corrupted every guest on the box,
including the protected `drill-r50` fixture. A 5 GiB `fallocate` probe moved guest `df` from
`977M used / 65G avail` to `6.0G / 60G` while thin-pool `data_percent` stayed **29.03 → 29.03** —
zero blocks allocated — and cleanup returned both to baseline. The floor reads `statfs`, which is
exactly what `fallocate` moves, so the condition it guards is genuinely present.
**Setup.** A **real** 2 GiB file in opengist's data volume (so its capture writes 2 GiB), then
`fallocate` to leave `69G / 63G used / 3.0G avail / 96%` — **both floor terms deliberately still
clear**, so the run would start. Capture order was established empirically from the previous run's log
(opengist first, privatebin second), not assumed.
**What happened, 06:40:03:**
```
Volume dump: opengist/opengist_opengist_data → 2.0 GB # unguarded
Volume dump: privatebin/privatebin_privatebin_data → 2.5 KB
App-data backup completed … 2 volume dump(s) (20.737s)
[WARN] Recovery unit capture REFUSED for opengist — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted
[WARN] Recovery unit capture REFUSED for privatebin — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted
[INFO] Event pushed: recovery_unit_capture_failed (error) — … ×2 (HTTP 200)
```
**What holds:** it refuses **per app** rather than aborting the run; **nothing was deleted** (both
units present afterwards); and the operator alert reached the hub — `recovery_unit_capture_failed`,
severity `error`, accepted `HTTP 200`, twice.
**What does not hold — measured, not inferred.** The refusal message claims *"the previous unit is
untouched"*:
| unit file | before | after |
|---|---|---|
| `privatebin/volume-dumps/privatebin_privatebin_data.tar` | `26c546c2…` | **`b538ab89…`** |
| `opengist/volume-dumps/opengist_opengist_data.tar` | 182,272 B | **2,147,666,432 B** |
Both were rewritten by the earlier leg, while each `manifest.json` kept
`created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so
a unit's payload can be swapped under a stale descriptor and nothing inside the unit can detect it.
**Cause, in the code and not the log.** The floor is consulted in exactly one place —
`m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which
writes a manifest and a compose copy: a few KB. `runVolumeDumps` (`backup.go:535`) — the leg that
writes the bulk, and the leg that consumed the reserve — has **no floor check at all**; its gates are
protected-stack, volume-less, disconnected, decommissioned. And it runs first *by design*
(`backup.go:483`). **The floor guards the cheap leg and not the leg that fills the volume.**
This is why R-165 is IMPLEMENTED and not PROVEN-LIVE: that row records B2 as the deliberate
replacement for the bulkhead the `mp1` partition provided, and pre-merge the unguarded leg could only
fill a dedicated 20 G partition — post-merge it can reach Docker's data-root. Filed as **R-181**.
**No code was written**, per the runbook's §7.
**Cleanup:** fill file and payload removed; `df` back to `1.2G used / 65G avail`; a clean re-run left
both units valid (`Volume dump … 178.0 KB` / `2.5 KB`, `App-data backup completed … (2.399s)`).
---
## 5. demo-felhom — the pipeline proof
**Install path — deliberately different, and this is the reason the second box exists.**
```
./felhom-host-install.sh --customer-id demo-felhom --mode appliance --vmid 9201 \
--cores 3 --memory 12288 \
--force-gitea-golden \
--passphrase-file /root/.pp-demo-felhom
```
A copy of golden 0.192.0 already sat on this box's `local` storage — it is where the golden was
**baked** at 06:58 (a `.log` beside it), sha `54e2a4c4…`. `--force-gitea-golden` is the documented
C.3 customer path and overrides local discovery in **both** pre-flight and step 7, which pre-flight
confirmed: `golden: none local — will fetch + verify from Gitea in step 7/8`. The bake artifact was
left untouched.
**`fetch_verify` succeeding against the vouched sha — the observable this box exists to produce:**
```
5/8 fetching agent binary v0.120.0 from Gitea …
verified sha256 a7763d31b55b5ce7… matches the hub manifest
7/8 fetching golden v0.192.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
verified sha256 54e2a4c431daf580… matches the hub manifest
golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
Day-0 provision SUCCESS — vmid=9201 host_id=demo-felhom-8363b5 customer=demo-felhom
```
Both artifacts — agent and golden — were fetched anonymously (the normal customer shape) and
sha-verified against the manifest. Controller `0.192.0` healthy.
**Layout:** `mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=250G`;
`pct config 9201 | grep -c '^mp1:'` → **0**. Both binds real mounts
(`…disk--1[/docker]`, `…disk--1[/sys_drive]`), `df` one figure — `246G 977M 233G 1%` on all three
paths — `stat -c %d` = `64519` on all three, fstab carries both binds.
**Reboots:** 1 — 09:18:55 → healthy 09:19:09; 2 — 09:19:12 → 09:19:26; 3 — 09:19:30 → 09:19:45. All
three paths still mountpoints after each; `uptime -s` = `2026-08-03 07:19:34 UTC`, matching reboot 3.
**Journey:** claim (`302 → /`, discriminator flipped to `authentication required`; code supplied by
the operator after a resend) → deploy `opengist` (`{"ok":true}`, `Up (healthy)`) → capture
(`Volume dump: opengist/… → 178.0 KB`, `Recovery unit captured`, unit on
`/dev/mapper/pve-vm--9201--disk--1`, marker present inside the tar) → **restore**
(`{"ok":true,"message":"opengist visszaállítva (primary)."}` in **9.4 s**, marker back with identical
sha256 `bc5507987f3f56dc19a9c24785826c304f1a986998d4aeb0b06d1a810b59e939`, app healthy).
**Ceiling:** 233 GiB available to a recovery unit, against the 45 GiB the pre-wipe `mp1` offered.
**Step 8 was NOT repeated here, deliberately** — stating it rather than leaving it ambiguous. The
floor fired on demo-hp and R-181 characterises it fully; re-firing would add no information and would
mean filling a 246 G volume.
---
## 6. Vouching
Already done before the session (§2a), at **07:23:26 CEST on 2026-08-03**, by an operator action in
the hub UI. The session's own manifest write was the **agent** half, at **07:44:31**:
`Artifact manifest set: agent=0.120.0 golden=0.192.0 min_agent="0.113.0" wrapper_sha=true`. The
manifest afterwards, read back: `agent_sha256=a7763d31b55b5ce7…10b9d`,
`golden_sha256=54e2a4c431daf580…43b3e0`, `min_agent=0.113.0`, `wrapper_sha256=104db0a4…` (preserved
verbatim). The R-120 gate did not block: golden 0.192.0 equals the newest controller the fleet
reports.
---
## 7. Teardown — all three layers
1. **The machine.** No throwaway guest was created this session, so there is none to delete. VM 300
`drill-r50` on demo-hp — the protected drift fixture — was never touched and is still `stopped`;
it is not in the `felhom` pool (`pvesh get /pools/felhom` listed only `lxc/9201`), so the
uninstall's shared-box logic never reached it.
2. **The host.** `local-lvm`, before → after: **demo-felhom 29.31 % → 1.37 %** (the old 200 G + 50 G
volumes returned; the new 250 G volume is thin and barely allocated). **demo-hp 39.13 % →
36.75 %** — *higher than a clean reinstall would leave it*, because the floor test's 2 GB tar
blocks cannot be reclaimed: `fstrim` inside an unprivileged LXC returns
`FITRIM ioctl failed: Operation not permitted`. No operational impact (the guest shows 65 G free of
69 G, the host 35.7 GiB free of 53.93), but it is real residue and is stated rather than rounded
away. Each box's data volume is now the only data volume; no old guest volumes remain.
**Residue found and cleared by hand on demo-hp:** `--uninstall` left the NAS network-storage units
`mnt-felhom\x2ddrives-Felhom\x2dShare.{mount,automount}` behind (automount `failed`, parent bind
still mounted) → **R-179**. demo-felhom left none, because it had no network share configured.
3. **The hub.** **No old records exist to dispose of, and this is the honest finding, not an
omission:** both enrollments were **idempotent** — `host REUSED (idempotent — existing
credential)`, `host_id: demo-hp-bb76ea` and `demo-felhom-8363b5`, the same ids as before. The
reinstalls therefore produced **no new host records**, so nothing was orphaned and nothing needed
deleting. Final register: `demo-felhom-8363b5 ONLINE 0.120.0`, `demo-hp-bb76ea ONLINE 0.120.0`,
`drill-r50-0a4f9a DOWN 0.113.0` — the same three rows as at P6. No scratch customers were created.
`/appliances` returns 404 on hub 0.89.0 — there is no appliance-record surface to clean.
**Secrets:** both retrieval passphrases were moved file→file into 0600 files, used via
`--passphrase-file`, and `shred -u`'d afterwards on both hosts along with the session cookie files and
helper scripts; the local scratch copies were deleted. Nothing was written to a committed file and
`curl -w '%{redirect_url}'` was never used (R-132).
---
## 8. Registers changed
| Row | Change |
| File | |
|---|---|
| **R-178** | **CLOSED** — both boxes reinstalled and proven, by two different supply paths |
| **R-165** | **IMPLEMENTED**, not PROVEN-LIVE — operator ruling; the layout half is proven, the B2 half is not (→ R-181) |
| **R-115** | **Third instance recorded** — agent 0.120.0 built, deployed to both hosts, never published |
| **R-181** *(new)* | The capture floor guards the recovery-unit leg and not `runVolumeDumps`; its "previous unit is untouched" claim measured false |
| **R-180** *(new)* | `--archive-storage` is not cross-checked against the ACL grant; the 403 lands at step 8/8, after root@pam has been rotated |
| **R-179** *(new)* | `--uninstall` leaves NAS network-storage systemd units behind when a share was configured |
| `controller/internal/backup/admission.go` | **new** — the gate, the memo, the estimator |
| `controller/internal/backup/admission_test.go` | **new** — 11 tests |
| `controller/internal/backup/backup.go` | run scope + gates in the DB and volume legs |
| `controller/internal/backup/recovery_unit.go` | `floorVerdict` size-aware; capture leg via `admitApp` |
| `controller/internal/backup/capture_floor_test.go` | 3 call sites updated for the new signature |
| `controller/README.md`, `REUSE.md`, `CHANGELOG.md` | |
| `app-catalog-felhom.eu/templates/papra/docker-compose.yml` | mount moved to `/app/app-data` |
**IDs established free before minting:** `grep -ro "R-179\b\|R-180\b\|R-181\b\|R-182\b"` over
`documentation/` and `*.md` → **0 hits**, and over all four repo roots (`felhom-agent`,
`felhom-controller`, `felhom.eu`, `app-catalog-felhom.eu`) → **0 hits**. `R-182` was checked and left
unused.
## 4. Tests — 28 packages `ok`, `rc=0` (read separately from any commit)
---
All 11 new tests pass, plus the pre-existing floor suite. Refusal assertions are **sha256 tree
fingerprints before and after**, never log lines — the defect being fixed *is* a log line the tree
contradicted.
## 9. CI — checked by run id, not assumed
The DB leg cannot run without Docker (`DiscoverDatabases` shells out), so its gate is pinned by an
**AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne`. `strings.Contains` is
insufficient: a commented-out call still contains the string.
Two commits, both docs-only.
### Red-proofs — each demonstrated failing, then restored
| repo | commit | CI run | result |
|---|---|---|---|
| `felhom-agent` | `9dfd89c` — *docs: agent 0.120.0 published + vouched…* | **run 46**, `head_sha 9dfd89cb` | **success** (`gates`) |
| `felhom.eu` | `aa62449` — *R-178 CLOSED: both demo boxes reinstalled…* | **run 47**, `head_sha aa624496` | **success** (`gates`) |
| # | Mutation | Result |
|---|---|---|
| 1 | **Both** dump-leg `admitApp` gates removed (= exactly v0.192.0) | Scenario A **RED** — *"the VOLUME leg ran for a refused app"*; with the leg assertions temporarily made non-fatal, the **tree fingerprint changed** too. Also red: Scenario C, Scenario D, and the AST wiring test (which named the DB leg specifically) |
| 2 | The entire size term removed from `floorVerdict` (both its thresholds) | Scenario D **RED** — 0 alerts where 1 was required |
| 3a | The reserve removed entirely | Scenario F **PASSED — recorded honestly.** The specified mutation does not exercise the assertion: removing the reserve makes every app write, which overwrites and adds but **deletes nothing**, so a deletion-watching test correctly stays green |
| 3b | A prune injected into the refusal path | Scenario F **RED** — this is the mutation that proves the test watches deletion |
| 4 | Floor moved above the warning band (90% / 6 GiB) | `TestFloorSitsBelowTheCriticalWarningBand` **RED** |
Queried with
`curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/<repo>/actions/tasks?limit=3"` and matched on
`head_sha`, per `CLAUDE.md`'s pull-check rule — CI mails on failure, which is a push signal; this is
the pull check that catches a lost or unread mail.
Every mutation removed **every** guard its test covers (#1 removed both dump-leg gates, not one).
**`--no-verify` was NOT used**, anywhere. This clone is armed (`core.hooksPath = .githooks`). Both
gate entry points were also run by hand before committing:
`felhom.eu/scripts/repo_gates.py` → **all five OK** (site, hostinstall, hub-confirm,
manifest-bearer, reuse-refs), and `felhom-agent/scripts/agent_gates.py` → **OK** (reuse-refs).
No test suite was run because no code changed in either repo.
## 5. Live validation — demo-hp guest 9201 (Tier 0), the method that found the defect
---
**Method:** endpoint-level — `POST /api/debug/backup/dbdump`, the exact endpoint the debug UI button
calls, which runs the production `RunDBDumps`. No browser on DooPlex.
## 10. Observations — noticed and NOT acted on
**The instrument was re-proven before use.** demo-hp's thin pool is 53.93 GiB, so a real fill of a
70 G volume would exhaust it and corrupt every guest. A 5 GiB `fallocate` step moved guest `df`
1.2G → 6.2G while thin-pool `data_percent` held **36.83 → 36.83** — zero blocks allocated. Re-checked
at every step of the fill.
- **The runbook's central claim was wrong in a way that mattered.** R-178 said *"a reinstall is now a
self-contained piece of work with no code left to write."* True about code; false about the artifact
channel — the agent half of the merge was unpublished, and following the documented path without
checking would have downgraded both boxes and produced a green, meaningless result. **No code change
was needed to make any step pass** — §10 asks this loudly, and the answer is no. What was needed was
a publish.
- **Prove-then-vouch was already spent when the session opened.** Not a defect in anything, but the
rule protected nothing because nothing enforced it. R-120's gate is the shape that would.
- **A per-stack `POST /stacks/<n>/backup` does not produce volume dumps** — the volume leg lives in the
full app-data pass. Not wrong, but the endpoint's name suggests otherwise and it cost time here.
- **privatebin's recovery unit carries no user data** (its catalog entry declares no `backup:`
section), so restoring it loses every paste. That may be intentional for an expiring, E2E-encrypted
paste bin — but nothing in the app's description tells the customer so. Catalog question, not filed.
- **V-c doubles systemd's mount-unit count** — every docker overlay appears twice,
`var-lib-docker-…-merged.mount` and `var-lib-felhom-docker-…-merged.mount`, because `/var/lib/docker`
is a bind of `/var/lib/felhom/docker`. Cosmetic, inherent to the chosen variant, no action.
- **`fstrim` cannot run inside the guest** (`EPERM`, unprivileged LXC), so space freed inside the guest
is not returned to the thin pool. It did not matter here; on a box that fills and empties repeatedly
it would.
- **The floor's own message mixes units** — it reports `64.2/68.7 GB used (93%)` against a threshold
stated as `97% used or 1.0 GiB free`, while `df` showed 96 %. GB-vs-GiB, so the percentage term
fires later than an operator reading `df` would expect. Minor; noted on R-181's fix shape rather
than filed separately.
### Headroom term — 08:59:46, 906 MB free / 99% used
| Observable | Result |
|---|---|
| Tree fingerprint before | `TREE_SHA=111d1760c18d3440f700634ab325f8b8` (10 files; opengist's tar **182,272 B** — R-181's own "before" figure) |
| Tree fingerprint after | **`111d1760c18d3440f700634ab325f8b8` — identical** |
| Volume dumps written | **0** (baseline run at 08:58 wrote 2) |
| `Stopping <app> for safe volume dump` | **absent** — and this is evidence, not an absence, because that line **is** present in the 08:58 baseline |
| Operator alerts | one `recovery_unit_capture_failed` per app, severity `error`, HTTP 200 |
Free space restored → re-run at **09:01:33**: both apps captured normally.
### Size term — 09:03:00, proven separately
Reproducing the original sequence: a real 2 GiB file planted in opengist's volume, backed up so its
**previous** tar became **2,147,666,432 B** (the exact live figure), then the filesystem set to
**91% used / 2.9 GB free — both headroom terms deliberately clear**.
- **opengist refused `(size)`** — *"this app's last backup was 2.0 GB and writing it again would cross the reserve"*
- **privatebin ADMITTED and dumped normally** — the term is per-app, not a global halt
- Tree unchanged; 1 volume dump instead of 2
### One honest correction to the "app not stopped" claim
`StartedAt` on both apps *did* move, 26 s **after** the refusal. It was the **quiesce loop** for the
whole-guest PBS backup, which my fill had broken — not the app-data path. Its own backoff logic then
behaved correctly (*"deferring its next quiesce by 15m so the apps are not stopped again for a backup
that cannot succeed"*). The app-data claim rests on the **absence of the `Stopping … for safe volume
dump` line**, which is the line that appears when that leg bounces an app.
## 6. The `du` measurement (§Part 1.3) — measured, then rejected
**66 timed runs** on demo-hp guest 9201, `docker run --rm -v <vol>:/v alpine du -sb /v`:
**median ~355 ms per volume, range 341–404 ms** — on volumes holding **tens of KB**. The cost is
container start-up, not the walk, so it does not shrink for small apps and only grows for real ones.
**Rejected**, on two grounds beyond the number: `docker run` needs the writable layer, so the
measurement mechanism can fail under exactly the disk pressure the reserve exists to handle; and the
previous-dump estimate measures the **artifact that will be written** rather than the live volume,
which is the truer predictor. The previous-dump estimate stands.
## 7. The refusal message as shipped, and what it guarantees
```
[WARN] [backup] App backup REFUSED for opengist (headroom) — refused: backing up this app would
leave the filesystem below the reserve (reserve: 97% used or 1.0 GiB free; the filesystem is already
below it, before this app's estimated 178.0 KB write) — /mnt/sys_drive: 64.3/68.7 GB used (94%),
0.9 GB free; NO database dump, NO volume dump and NO recovery-unit capture was written for it, the
previous unit is untouched and NOTHING was deleted
```
**It guarantees, for that app in that run:** no DB dump, no volume dump and no capture were written;
every file under `backups/primary/<app>` is byte-identical; the app was not stopped; nothing anywhere
was deleted; exactly one operator alert was sent. All five verified by fingerprint above.
**The wording was not weakened to fit the behaviour** — the behaviour moved so the wording became
true. What was *added* is the bound term (`headroom` / `size`) and the estimate.
**v0.193.1 — found by this very proof run.** The estimate was rendered fixed to two-decimal GiB, so
opengist's real **178 KB** printed as `estimated 0.00 GiB write`, which reads as *no estimate was
available* — the opposite of what happened. Shipped the same session because it is the same defect
class the whole task is about. Re-verified live after redeploy: `estimated 178.0 KB write`.
## 8. papra (R-156, last leg)
**Precondition checked, not inherited** — both boxes were wiped and rebuilt today, so the 2 August
evidence was re-measured: `docker ps -a` (**including stopped**) on **both** demo guests → no papra;
hub `/hosts` → exactly two enrolled hosts (`demo-felhom-8363b5`, `demo-hp-bb76ea`), **zero** papra.
**Decided from the image, not the README:** `WORKDIR=/app`, `DATABASE_URL=file:./app-data/db/db.sqlite`,
`DOCUMENT_STORAGE_FILESYSTEM_ROOT=./app-data/documents`, `PAPRA_CONFIG_DIR=./app-data` — and
**`/app/data` does not exist in the image at all**.
**Departure from the task's stated preference order, stated because it was deliberate.** Option (1)
(reconfigure the app to write to `/app/data`) *was* available — all three paths are env-settable. Not
taken: it enumerates data paths, so a fourth added upstream would silently escape to the writable
layer again — this defect re-armed and invisible. Mounting the app's own data **root** captures every
current and future path by construction.
**Gate output — the arbiter, run in both directions:**
- fixed → `papra CLEAN`, with the self-test passing on that run: *"prober flags the R-156 signature and clears a correct template — trustworthy"*
- reverted to `/app/data` (red-proof on the **real template**, not just the canary) → `BROKEN`: *"mount /app/data is NOT writable by the app's own uid=999"*, *"DATA in the writable layer at /app/app-data/db (db_signature=True, e.g. ['db.sqlite'])"*, *"declared volume /app/data is EMPTY"*
- `catalog_gates.py papra` (full, not `--fast`) → **rc=0**, all three gates OK
**Two operational findings about the gate:** it needs **root** (it reads `/var/lib/docker/volumes`,
mode `drwx--x---`; as a normal user its own canary fails UNDETERMINED and it correctly refuses a
verdict — fail-closed working as designed), and it hardcodes scratch path `/srv/felhom-gate`, created
on DooPlex. Unscoped it deploys all 53 templates; that run was aborted after 10 minutes and its
`volgate-*` scratch projects were cleaned up.
## 9. §3's correction — confirmed in passing, not chased
`restore_points.go:57-59` takes the manifest's mtime and then `newestArtifact` over the `.sql` and
`.tar` files, so **the newest of the three wins**. The restore point does **not** show a stale
timestamp. Confirmed and dropped, as instructed.
## 10. Register
| ID | Change |
|---|---|
| **R-181** | **CLOSED — SHIPPED** (v0.193.0 + v0.193.1), with the live evidence above |
| **R-156** | **CLOSED** — all three apps fixed |
| **R-110** | WAITING-ON-OPERATOR → **READY**, ruling attached: **option (b), tag-tracked**, and it must cover **both** channels (the `/scripts/` git-sync *and* the nine files fetched from `raw/branch/main`) or it only half-works |
| **R-115** | WAITING-ON-OPERATOR → **READY**, ruling attached: **mechanism (b)**, a build-side gate refusing to deploy or vouch an unpublished version; the third instance (agent v0.120.0) would have silently downgraded both demo boxes while reporting success |
| **R-182** | **NEW.** ID established free: `grep -ro "R-182\b"` over `documentation/` and `*.md` → 2 hits, both prose in `REPORT.md` recording it as *"checked and left unused"*; `R-183` → 0 hits and remains free |
**R-165** is collapsed to CLOSED/PROVEN-LIVE in `ROADMAP.md`; the capability map's local-backup row
moves to **PROVEN-LIVE, both halves**, because the live fill proved the fixed behaviour for **both**
reserve terms.
## 11. Observations — noticed, documented, NOT acted on
1. **R-182 (filed).** The periodic status refresh (`GetFullStatus` → `captureAllRecoveryUnits`) runs
with no admission scope, so a refused app re-alerts on every poll — measured live: a second
identical alert pair 13 s after the run's. **Pre-existing in v0.192.0**; R-181 changed neither
caller. Its mitigation is a *comment* claiming the hub owns cooldown — which is exactly the
"invariant asserted in a comment with no test pinning it" shape, so verify at the hub before
scoping.
2. **A reserve refusal does not make the run fail.** The DB and volume legs record `SKIP`, not `FAIL`,
so `lastDBDump.Success` stays true and the customer-facing status does not turn red. Deliberate and
consistent with v0.192.0 (the capture refusal never set it either), and the operator alert is the
signal — but it means "backup succeeded" and "every app was backed up" are not the same statement.
3. **`UnitSpace.UsedPercent` and `df` disagree** — `df` reported 99% where the alert said 94%, because
`df`'s figure accounts for ext4 reserved blocks and the floor's does not. Harmless here (the
free-byte term bound), but a percent-term threshold is being compared against a number the operator
cannot reproduce with `df`.
4. **The whole-guest PBS backup fails when the volume is near-full**, pushing
`whole_guest_backup_failed` (severity `error`). Expected under a deliberate fill, and its backoff
behaved correctly; noted because it is collateral any future fill test will also produce.
## 12. Teardown
Fill file removed; the planted 2 GiB file removed; a final backup regenerated a correct 178 KB tar;
`pct fstrim 9201` returned 67.5 GiB and the thin pool settled at **29.43%**, *below* its 36.83%
baseline. The backups tree is byte-identical to the pre-test fingerprint. Guest helper scripts and the
credential file `shred`-ed. `volgate-*` scratch compose projects removed; the unrelated 9-day-old
`jarr-*` containers on DooPlex were left untouched. papra is **not** left deployed.
No `--no-verify` was used on any push; the `felhom-controller` pre-push hook ran and reported
`gates OK` on both pushes.