admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: R-110 + R-115 closed, R-182 re-scoped by measurement, R-183/R-184 filed
gates / gates (push) Successful in 7s
Details

Browse Source 
R-182's direction REVERSED by Part 0's measurement. Filed yesterday as "the
reserve re-alerts on every status refresh" — too many alerts, seen at the
sending end. Measured at the receiving end: 9 events received today, 2 operator
emails sent. When two apps are refused in the same second the operator is told
about ONE; the other is dropped before LogNotification, so it leaves no row on
any channel and cannot be audited. The operator cooldown key is
customerID:eventType(+tier) and the capture-failed event carries `app` but no
`tier`, so the key has no app identifier. Same failure mode as R-97a, in a
second event type that never opted into the narrow fix. Nothing changed —
Part 0 was investigation only.

Correction owed: yesterday's report said "one recovery_unit_capture_failed per
app, HTTP 200". True of what the CONTROLLER pushed; a reader would take it as
"the operator was told about each app", which is false.

R-110 CLOSED (installer v1.23.0). Both channels moved. The spec's mechanism for
channel 2 rested on a factual error — the run-time fetches are sixteen, not
nine, and come from felhom-agent, not this repo — so no tag here could cover
them; pinned to the agent version being installed instead, on the operator's
ruling. Channel 3 needed no change: the URL never carried a ref, so no hub
change and no hub bump.

R-115 CLOSED. release-agent.sh builds, tags, publishes and verifies by an
independent download; check-published-versions.py refuses a tag with no package;
CI now runs the full gate set so it actually runs.

R-183 NEW+CLOSED: a fresh install fetched the vouched agent binary and its
sixteen config files from two different refs, and nothing compared them.

R-184 NEW: nothing stops the hub vouching a version that was never released.
The R-115 gate cannot see it — measured, the hub manifest and Gitea's package
listing are both 401 anonymously.

capability map: new PROVEN-LIVE row for the published installer channel.
STATUS.md 138 -> 127 lines.
This commit is contained in:
authentik Default Admin
2026-08-03 12:44:08 +02:00
parent e79a20bbed
commit b0b269b28d
7 changed files with 274 additions and 229 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+171 -187
View File
@@ -1,216 +1,200 @@
# REPORT — R-181 (the reserve guards the write that fills the disk) + R-156 (papra) + two operator rulings
# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed
**Date:** 2026-08-03 · **Repos:** `felhom-controller` (v0.192.0 → **v0.193.1**), `app-catalog-felhom.eu`, `felhom.eu` (docs only — **no hub change, no hub version bump**)
**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**)
**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.**
## 1. Baselines — re-read on arrival, all matched §1
## 1. Baselines — re-read on arrival, both matched §1
| Repo | `main` @ arrival | Version | Shipped |
| Repo | @ arrival | Version | Result |
|---|---|---|---|
| `felhom-controller` | `4be6467b501b` | v0.192.0 | **v0.193.0 `fef07c3`** → **v0.193.1 `6c43bf6`** |
| `app-catalog-felhom.eu` | `7cb58ecdf8e7` | n/a | `122bbee` |
| `felhom.eu` | `6b5d64c1fa73` | hub v0.89.0 | docs only, **no bump** |
| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` |
| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only |
All three clean (`git status --porcelain` empty, `HEAD == origin/main`) before every build.
## 2. Part 0 — the R-182 measurement, and it REVERSED the row
## 2. The fix
Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed
at the sending end. Measured at the **receiving end**, it is the opposite.
**One admission verdict per app per run** (`controller/internal/backup/admission.go`), taken before
that app's **first** write and consulted by all three legs — DB dump, volume dump, unit capture. The
three write under one per-app root (`appbackup.RecoveryUnitPath`), which is what makes one verdict
able to cover them honestly.
Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db`
alone would have read stale data, the exact trap this project recorded before), freshness confirmed by
the newest `notification_log` row post-dating the session.
- **Lazy, not run-wide.** App A's dump can put app B under the reserve; a run-start verdict reads a
disk that no longer exists. **Never re-decided between an app's own legs** — that is the split being
closed. **Reset per run.**
- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act, so a refused app is never
bounced. **After** the volume-less check, which has no write to gate.
- **Exactly one operator alert per refused app per run.** Leg order unchanged.
- **Size term added:** *would this app's write cross the reserve?* — estimated from its previous
`.sql` + `.tar`. **No history → headroom-only**, or the first backup becomes the one that can never
happen; the alert says so when that applies.
**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.**
## 3. Files
| time | apps refused (events in) | operator emails out |
|---|---|---|
| 06:40:03 | privatebin, opengist | **opengist only** |
| 08:59:46/47 | opengist, privatebin | **privatebin only** |
| 08:59:59 | privatebin, opengist | **none** |
| 09:03:00 | opengist | **none** |
| 09:07:06 | privatebin, opengist | **none** |
| File | |
**Cause, confirmed at source:** the operator cooldown key is
`customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded).
`RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key
holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the
next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any
channel** and cannot be audited afterwards.
This is **R-97a's failure mode in a second event type**; that row's own comment states it
(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was
written narrow on purpose; `recovery_unit_capture_failed` simply never opted in.
**A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app,
HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the
operator was told about each app* — which is false. The gap between an accepted event and a sent
email is the whole of this row.
**Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape.
## 3. Probes
| | Question | Method | Verdict |
|---|---|---|---|
| **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) |
| **P-B** | does Gitea serve `raw/tag/<tag>/<path>`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` |
| **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** |
**Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in
~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone
(`git ls-remote --tags` → 0 at the time).
## 4. §8.2's three channels — enumerated
| Channel | Before | After | |
|---|---|---|---|
| 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs |
| 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below |
| 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission |
**Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.**
There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so
no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"*
was impossible for them. Raised before building; operator ruled to pin them to **the agent version
being installed**, which the installer already resolves from the hub manifest and already sha-verifies.
That is strictly better than any installer-derived tag: binary and configs now come from one ref.
**Channel 3 needed no change because the URL never carried a ref** —
`https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template
change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose.
## 5. The tag convention
- **Shape:** `installer-v<SCRIPT_VERSION>` in `felhom.eu` (prefixed so it cannot be read as a hub,
agent, controller or golden version); `v<semver>` in `felhom-agent` (that repo versions one thing).
**No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time,
and channel 1's ref lives only in the manifest.
- **Publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (sidecar *and*
init container), commit, sync.
- **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**.
## 6. Scenario A — proven by HTTP
A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and
three sync periods were allowed to pass so "unchanged" means "had every chance to change":
```
website tree (main): .worktrees/6a82719… <- ADVANCED to the new commit
scripts tree (tag): .worktrees/bee6848… <- STAYED
sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
sha256 after push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
marker present at the served URL? 0
https://felhom.eu/ -> HTTP 200
```
Both halves of the split in one observation: the site still tracks `main`, the installer does not.
## 7. Scenario B — publish and rollback, both directions
| act | result |
|---|---|
| `controller/internal/backup/admission.go` | **new** — the gate, the memo, the estimator |
| `controller/internal/backup/admission_test.go` | **new** — 11 tests |
| `controller/internal/backup/backup.go` | run scope + gates in the DB and volume legs |
| `controller/internal/backup/recovery_unit.go` | `floorVerdict` size-aware; capture leg via `admitApp` |
| `controller/internal/backup/capture_floor_test.go` | 3 call sites updated for the new signature |
| `controller/README.md`, `REUSE.md`, `CHANGELOG.md` | |
| `app-catalog-felhom.eu/templates/papra/docker-compose.yml` | mount moved to `/app/app-data` |
| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** |
| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** |
## 4. Tests — 28 packages `ok`, `rc=0` (read separately from any commit)
`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved
to `main`'s head — a **byte no-op**, verified by the served sha not changing.
All 11 new tests pass, plus the pre-existing floor suite. Refusal assertions are **sha256 tree
fingerprints before and after**, never log lines — the defect being fixed *is* a log line the tree
contradicted.
## 8. Files, commits, tags
The DB leg cannot run without Docker (`DiscoverDatabases` shells out), so its gate is pinned by an
**AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne`. `strings.Contains` is
insufficient: a commented-out call still contains the string.
**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b`
(marker removed), plus the docs commit below.
`scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` ·
`manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` ·
`documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md`
### Red-proofs — each demonstrated failing, then restored
**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT).
`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** ·
`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md`
**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and
`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from;
`configs/` is byte-identical there and at `main`, so nothing depended on the choice).
## 9. Tests and red-proofs
| Check | Result |
|---|---|
| `felhom.eu` `repo_gates.py --fast` | all 5 gates OK |
| `felhom-agent` `go build ./... && go vet ./...` | OK |
| `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
| `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) |
| `agent_gates.py` (full) | both OK |
**Red-proofs, each demonstrated failing then restored:**
| # | Mutation | Result |
|---|---|---|
| 1 | **Both** dump-leg `admitApp` gates removed (= exactly v0.192.0) | Scenario A **RED** — *"the VOLUME leg ran for a refused app"*; with the leg assertions temporarily made non-fatal, the **tree fingerprint changed** too. Also red: Scenario C, Scenario D, and the AST wiring test (which named the DB leg specifically) |
| 2 | The entire size term removed from `floorVerdict` (both its thresholds) | Scenario D **RED** — 0 alerts where 1 was required |
| 3a | The reserve removed entirely | Scenario F **PASSED — recorded honestly.** The specified mutation does not exercise the assertion: removing the reserve makes every app write, which overwrites and adds but **deletes nothing**, so a deletion-watching test correctly stays green |
| 3b | A prune injected into the refusal path | Scenario F **RED** — this is the mutation that proves the test watches deletion |
| 4 | Floor moved above the warning band (90% / 6 GiB) | `TestFloorSitsBelowTheCriticalWarningBand` **RED** |
| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired |
| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it |
| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state |
| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 |
| F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 |
Every mutation removed **every** guard its test covers (#1 removed both dump-leg gates, not one).
**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit*
`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also
retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was
this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API
token, and the only credential available (`~/.docker/config.json`) is a registry password that the API
rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.
## 5. Live validation — demo-hp guest 9201 (Tier 0), the method that found the defect
## 10. No version bumps, nothing built
**Method:** endpoint-level — `POST /api/debug/backup/dbdump`, the exact endpoint the debug UI button
calls, which runs the production `RunDBDumps`. No browser on DooPlex.
`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file
touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's
no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or
golden was built.
**The instrument was re-proven before use.** demo-hp's thin pool is 53.93 GiB, so a real fill of a
70 G volume would exhaust it and corrupt every guest. A 5 GiB `fallocate` step moved guest `df`
1.2G → 6.2G while thin-pool `data_percent` held **36.83 → 36.83** — zero blocks allocated. Re-checked
at every step of the fill.
## 11. Register
### Headroom term — 08:59:46, 906 MB free / 99% used
| Observable | Result |
| ID | Outcome |
|---|---|
| Tree fingerprint before | `TREE_SHA=111d1760c18d3440f700634ab325f8b8` (10 files; opengist's tar **182,272 B** — R-181's own "before" figure) |
| Tree fingerprint after | **`111d1760c18d3440f700634ab325f8b8` — identical** |
| Volume dumps written | **0** (baseline run at 08:58 wrote 2) |
| `Stopping <app> for safe volume dump` | **absent** — and this is evidence, not an absence, because that line **is** present in the 08:58 baseline |
| Operator alerts | one `recovery_unit_capture_failed` per app, severity `error`, HTTP 200 |
| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed |
| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) |
| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described |
| **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs |
| **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released |
Free space restored → re-run at **09:01:33**: both apps captured normally.
**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each;
all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0
hits anywhere and remains free.
### Size term — 09:03:00, proven separately
## 12. Observations — noticed, documented, NOT acted on
Reproducing the original sequence: a real 2 GiB file planted in opengist's volume, backed up so its
**previous** tar became **2,147,666,432 B** (the exact live figure), then the filesystem set to
**91% used / 2.9 GB free — both headroom terms deliberately clear**.
1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs
either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails
closed where the mistake is made, needs no new credential).
2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`,
so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside
R-182 because it is what made that row take a day to get the right way round.
3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it
means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a
network gate is ever added there, that workflow needs the same change the agent's just got.
- **opengist refused `(size)`** — *"this app's last backup was 2.0 GB and writing it again would cross the reserve"*
- **privatebin ADMITTED and dumped normally** — the term is per-app, not a global halt
- Tree unchanged; 1 volume dump instead of 2
## 13. Teardown
### One honest correction to the "app not stopped" claim
`StartedAt` on both apps *did* move, 26 s **after** the refusal. It was the **quiesce loop** for the
whole-guest PBS backup, which my fill had broken — not the app-data path. Its own backoff logic then
behaved correctly (*"deferring its next quiesce by 15m so the apps are not stopped again for a backup
that cannot succeed"*). The app-data claim rests on the **absence of the `Stopping … for safe volume
dump` line**, which is the line that appears when that leg bounces an app.
## 6. The `du` measurement (§Part 1.3) — measured, then rejected
**66 timed runs** on demo-hp guest 9201, `docker run --rm -v <vol>:/v alpine du -sb /v`:
**median ~355 ms per volume, range 341–404 ms** — on volumes holding **tens of KB**. The cost is
container start-up, not the walk, so it does not shrink for small apps and only grows for real ones.
**Rejected**, on two grounds beyond the number: `docker run` needs the writable layer, so the
measurement mechanism can fail under exactly the disk pressure the reserve exists to handle; and the
previous-dump estimate measures the **artifact that will be written** rather than the live volume,
which is the truer predictor. The previous-dump estimate stands.
## 7. The refusal message as shipped, and what it guarantees
```
[WARN] [backup] App backup REFUSED for opengist (headroom) — refused: backing up this app would
leave the filesystem below the reserve (reserve: 97% used or 1.0 GiB free; the filesystem is already
below it, before this app's estimated 178.0 KB write) — /mnt/sys_drive: 64.3/68.7 GB used (94%),
0.9 GB free; NO database dump, NO volume dump and NO recovery-unit capture was written for it, the
previous unit is untouched and NOTHING was deleted
```
**It guarantees, for that app in that run:** no DB dump, no volume dump and no capture were written;
every file under `backups/primary/<app>` is byte-identical; the app was not stopped; nothing anywhere
was deleted; exactly one operator alert was sent. All five verified by fingerprint above.
**The wording was not weakened to fit the behaviour** — the behaviour moved so the wording became
true. What was *added* is the bound term (`headroom` / `size`) and the estimate.
**v0.193.1 — found by this very proof run.** The estimate was rendered fixed to two-decimal GiB, so
opengist's real **178 KB** printed as `estimated 0.00 GiB write`, which reads as *no estimate was
available* — the opposite of what happened. Shipped the same session because it is the same defect
class the whole task is about. Re-verified live after redeploy: `estimated 178.0 KB write`.
## 8. papra (R-156, last leg)
**Precondition checked, not inherited** — both boxes were wiped and rebuilt today, so the 2 August
evidence was re-measured: `docker ps -a` (**including stopped**) on **both** demo guests → no papra;
hub `/hosts` → exactly two enrolled hosts (`demo-felhom-8363b5`, `demo-hp-bb76ea`), **zero** papra.
**Decided from the image, not the README:** `WORKDIR=/app`, `DATABASE_URL=file:./app-data/db/db.sqlite`,
`DOCUMENT_STORAGE_FILESYSTEM_ROOT=./app-data/documents`, `PAPRA_CONFIG_DIR=./app-data` — and
**`/app/data` does not exist in the image at all**.
**Departure from the task's stated preference order, stated because it was deliberate.** Option (1)
(reconfigure the app to write to `/app/data`) *was* available — all three paths are env-settable. Not
taken: it enumerates data paths, so a fourth added upstream would silently escape to the writable
layer again — this defect re-armed and invisible. Mounting the app's own data **root** captures every
current and future path by construction.
**Gate output — the arbiter, run in both directions:**
- fixed → `papra CLEAN`, with the self-test passing on that run: *"prober flags the R-156 signature and clears a correct template — trustworthy"*
- reverted to `/app/data` (red-proof on the **real template**, not just the canary) → `BROKEN`: *"mount /app/data is NOT writable by the app's own uid=999"*, *"DATA in the writable layer at /app/app-data/db (db_signature=True, e.g. ['db.sqlite'])"*, *"declared volume /app/data is EMPTY"*
- `catalog_gates.py papra` (full, not `--fast`) → **rc=0**, all three gates OK
**Two operational findings about the gate:** it needs **root** (it reads `/var/lib/docker/volumes`,
mode `drwx--x---`; as a normal user its own canary fails UNDETERMINED and it correctly refuses a
verdict — fail-closed working as designed), and it hardcodes scratch path `/srv/felhom-gate`, created
on DooPlex. Unscoped it deploys all 53 templates; that run was aborted after 10 minutes and its
`volgate-*` scratch projects were cleaned up.
## 9. §3's correction — confirmed in passing, not chased
`restore_points.go:57-59` takes the manifest's mtime and then `newestArtifact` over the `.sql` and
`.tar` files, so **the newest of the three wins**. The restore point does **not** show a stale
timestamp. Confirmed and dropped, as instructed.
## 10. Register
| ID | Change |
|---|---|
| **R-181** | **CLOSED — SHIPPED** (v0.193.0 + v0.193.1), with the live evidence above |
| **R-156** | **CLOSED** — all three apps fixed |
| **R-110** | WAITING-ON-OPERATOR → **READY**, ruling attached: **option (b), tag-tracked**, and it must cover **both** channels (the `/scripts/` git-sync *and* the nine files fetched from `raw/branch/main`) or it only half-works |
| **R-115** | WAITING-ON-OPERATOR → **READY**, ruling attached: **mechanism (b)**, a build-side gate refusing to deploy or vouch an unpublished version; the third instance (agent v0.120.0) would have silently downgraded both demo boxes while reporting success |
| **R-182** | **NEW.** ID established free: `grep -ro "R-182\b"` over `documentation/` and `*.md` → 2 hits, both prose in `REPORT.md` recording it as *"checked and left unused"*; `R-183` → 0 hits and remains free |
**R-165** is collapsed to CLOSED/PROVEN-LIVE in `ROADMAP.md`; the capability map's local-backup row
moves to **PROVEN-LIVE, both halves**, because the live fill proved the fixed behaviour for **both**
reserve terms.
## 11. Observations — noticed, documented, NOT acted on
1. **R-182 (filed).** The periodic status refresh (`GetFullStatus` → `captureAllRecoveryUnits`) runs
with no admission scope, so a refused app re-alerts on every poll — measured live: a second
identical alert pair 13 s after the run's. **Pre-existing in v0.192.0**; R-181 changed neither
caller. Its mitigation is a *comment* claiming the hub owns cooldown — which is exactly the
"invariant asserted in a comment with no test pinning it" shape, so verify at the hub before
scoping.
2. **A reserve refusal does not make the run fail.** The DB and volume legs record `SKIP`, not `FAIL`,
so `lastDBDump.Success` stays true and the customer-facing status does not turn red. Deliberate and
consistent with v0.192.0 (the capture refusal never set it either), and the operator alert is the
signal — but it means "backup succeeded" and "every app was backed up" are not the same statement.
3. **`UnitSpace.UsedPercent` and `df` disagree** — `df` reported 99% where the alert said 94%, because
`df`'s figure accounts for ext4 reserved blocks and the floor's does not. Harmless here (the
free-byte term bound), but a percent-term threshold is being compared against a number the operator
cannot reproduce with `df`.
4. **The whole-guest PBS backup fails when the volume is near-full**, pushing
`whole_guest_backup_failed` (severity `error`). Expected under a deliberate fill, and its backoff
behaved correctly; noted because it is collateral any future fill test will also produce.
## 12. Teardown
Fill file removed; the planted 2 GiB file removed; a final backup regenerated a correct 178 KB tar;
`pct fstrim 9201` returned 67.5 GiB and the thin pool settled at **29.43%**, *below* its 36.83%
baseline. The backups tree is byte-identical to the pre-test fingerprint. Guest helper scripts and the
credential file `shred`-ed. `volgate-*` scratch compose projects removed; the unrelated 9-day-old
`jarr-*` containers on DooPlex were left untouched. papra is **not** left deployed.
No `--no-verify` was used on any push; the `felhom-controller` pre-push hook ran and reported
`gates OK` on both pushes.
Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag
`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from
`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl
pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.