register: file R-156..R-164 in one pass, ranked; and record what mp1 is actually for
Nine rows into OPEN-ITEMS.md and ROADMAP.md, matching each file's column shape. R-156 and R-157 had lived only in audit documents - the identical "minted in a spike doc and never carried across" failure the register already records for R-153/R-154/R-155, caught by the catalog sweep's own section 8.0 while it was happening. R-158 was minted by a second session the same day for an unrelated finding, which is why the sweep's proposals were renumbered R-159..R-162 at filing time. All nine IDs verified free in BOTH backlog files before use. Part 0 settled the question the sizing item depended on, by reading: mp1 is RETENTION, not staging, and neither of the two framings was right. A unit is the KEPT copy on the app's OWN drive (backup.go:245-255); for an app with no HDD_PATH the namespace falls back to the system SSD - "the SSD-only system-data fallback" (appbackup/paths.go:26-27). There is no post-copy deletion: the only prune is F5 residue-on-old-drives when an app MOVES (backup.go:1053-1112). So mp1 retains the units of driveless apps only - not every app, but not transient either. Confirmed against the spike: sys_drive held exactly the four driveless apps and not calibre-web, which had a drive and was still backed up. A unit is volume tars + DB dumps only, never mp8 userdata (recovery_unit.go:20-25), so a 1 TB photo library can never overflow one. And mp1 gates the WHOLE chain, not just Tier 1: Tier-2 mirrors the unit "(always)" from RecoveryUnitPath (tier2.go:302,368) and Tier-3 carries it, so a unit that cannot be written leaves both with nothing to copy. Part 2 fired on both triggers - retention, and the fallback undocumented - so 07-backup-architecture.md gains section 7.5. Section 6.1 said a unit lives "on the app's own drive", which is true and was the whole story only for drive-resident apps; the no-drive case was undocumented, as was the sizing constraint. 7.5 records the mp0-50G-vs-mp1-20G mismatch, the measured ratios (DB app up to ~2x, 21.1GB -> 40.2GB; file-only 1.00x), and the bound this puts on D5's Lane-1 independence: restorable from the drive alone only while the unit still fits - about 19 GB file-only, about 10 GB DB-backed. No number proposed; the ratio is the operator's ruling (R-163). R-159/R-160 marked SHIPPED only after verifying the template changes are in app-catalog origin/main, and R-156's gate likewise (check-volume-persistence.py present). papra is NOT fixed - referred - so R-156 stays open on that one app. Ranked, with one line of reasoning each: R-157 first (an app can stay down indefinitely with mechanism B silent on every channel), then R-161 (the gate exists and nothing runs it, which is why R-156's class recurs - R-29's record is three orphaned gates and one enforced), R-156, R-163, R-158, R-164, R-162.
This commit is contained in:
@@ -484,6 +484,58 @@ Until D5 is actually implemented, §7.1's chain stands as the model.
|
||||
|
||||
---
|
||||
|
||||
### 7.5 Where a recovery unit is RETAINED — and the size bound it puts on Lane 1
|
||||
|
||||
Added 2026-08-02 from `audits/SPIKE-recovery-unit-space-2026-08-02.md` and
|
||||
`audits/CAMPAIGN-10-closeout-2026-08-02.md`. §6.1 says a Tier-1 unit lives *"on the app's own drive"*,
|
||||
which is true and is the whole story for a drive-resident app. **It is not the whole story for an app
|
||||
with no data drive, and that case was undocumented until now.**
|
||||
|
||||
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go:245-255`)
|
||||
returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what
|
||||
`internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit
|
||||
after it is copied onward: the only prune is F5 (`backup.go:1053-1112`), which removes residue on an
|
||||
**old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy
|
||||
indefinitely.
|
||||
|
||||
**What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`):
|
||||
`compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains
|
||||
`mp8` / `HDD_PATH` userdata** — a photo library on a 1 TB drive is not in a unit, and cannot make one
|
||||
overflow.
|
||||
|
||||
**The resulting mismatch, on a default appliance:**
|
||||
|
||||
| | size | holds |
|
||||
|---|---|---|
|
||||
| `mp0` `/var/lib/docker` | **50 G** | every app's live volumes — the thing a unit copies |
|
||||
| `mp1` `/mnt/sys_drive` | **20 G** | the retained units of every **driveless** app |
|
||||
|
||||
A DB-backed app's unit is up to **~2×** its data, because it carries the volume tar **and** the SQL
|
||||
dump (measured: 21.1 GB of app data → a **40.2 GB** unit). A file-only app's is **1.00×** (measured:
|
||||
homebox 2305 MB → 2305 MB). `--sysdata-grow` (`felhom-agent/cmd/felhom-agent/main.go:178`) defaults to
|
||||
**0**, is not derived from the physical drive, and no observed box passes it — demo-hp's production
|
||||
guest 9201 ships `mp0 50G / mp1 20G`.
|
||||
|
||||
**`mp1` gates the entire app-data chain, not just Tier 1.** Tier-2 mirrors the unit *"(always)"* from
|
||||
`RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too, so a unit that
|
||||
cannot be written to `mp1` leaves Tier-2 and Tier-3 with nothing to copy.
|
||||
|
||||
**The bound this places on Lane 1 (§3).** Lane 1's promise is that a customer can restore an app from
|
||||
the drive alone, without the operator and without the guest — D5 (§7.4) made that true by putting the
|
||||
portable secrets in the unit. **That independence is bounded by app size**, and the bound is:
|
||||
|
||||
> On a default appliance, an app can be restored by Lane 1 only while its recovery unit still fits the
|
||||
> retained space — **≈ 19 GB of app data for a file-only app, ≈ 10 GB for a DB-backed one.**
|
||||
> Past that the unit cannot be captured, and the app falls back to Lane 2's operator-driven
|
||||
> whole-guest route.
|
||||
|
||||
Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling**
|
||||
(**R-163**) — this section states the constraint, not a number. And **the same-device placement is
|
||||
intended, not a defect**: a driveless app's unit sits on the same SSD as its volumes, but Tier-2's
|
||||
cross-drive mirror, the whole-guest tier (which covers `rootfs + mp0 + mp1`, §6.1) and Tier-3 all cover
|
||||
device loss. What was missing was that this case existed at all, and that nothing warns when an app
|
||||
crosses the line — **R-158**.
|
||||
|
||||
## 8. The failure → recovery matrix
|
||||
|
||||
**This is the core artifact.** One row per failure. It is authoritative for recovery routes;
|
||||
|
||||
@@ -84,9 +84,19 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-137** | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with "<domain>"` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC |
|
||||
| **R-138** | **A shared-zone `cf_api_token` is a zone-wide DNS-write capability on a customer's box** — written 0600 to `/opt/docker/stacks/traefik/.env` (`controller/internal/infra/infra.go:123`) | READY (S) | — | Today each box holds a token for a zone nobody else uses, so the blast radius is one customer. Under a shared customer zone, one compromised tester box could repoint every other tester's DNS. The ACME path is already switchable — an empty token selects HTTP-01 (`traefik.yml.tmpl`) — so the fix is policy plus a guard that refuses to hand a shared-zone customer a zone-scoped token. Same audit §5.2 | CC |
|
||||
| **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC |
|
||||
| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. Next action is papra's referral only. See R-161 (nothing runs the gate) and R-159/R-160 | CC |
|
||||
| **R-157** | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms, and a settle-condition fix closes only one.** **(A)** the sweep runs ~5 s after controller start (`cmd/controller/main.go:236`) while docker is still restoring, sees "no boot-orphaned apps", and never re-checks — **intermittent, 3 of 6 hard resets**; the predicate itself is sound (`bootrecon.go:100`, and `IsDownState` includes `StateDegraded`, `manager.go:55`). **(B)** an interruption that lands mid-compose leaves the stack with **zero containers**, which is exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp check reported **`0 currently down`** while a `deployed: true` app was not running: **silent on every channel**. | **READY (M)** | — | **Two fixes, not one.** A: a settle condition before the single observation, or one re-evaluation at the end of `deadAppBootGrace`, still bounded to the same attempts. B: needs a signature that distinguishes "customer stopped this" from "interrupted between containers" — B is F-CRIT-1's shape (`CLAUDE.md` false-invariant #4) reached via hard reset rather than quiesce. Test the **consequence** (app running again before the grace expires), not the mechanism. Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` | CC |
|
||||
| **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC |
|
||||
| **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `<project>_<name>` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC |
|
||||
| **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |
|
||||
| **R-161** | **The volume-persistence gate exists and nothing automated runs it.** `check-volume-persistence.py` is enforced by **convention**: the catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — none checked, none exists). | **BLOCKED** — needs a decision, not a task | a runner, or a cross-repo design ruling | This project's own record says convention is not enough: **R-29 has three orphaned gates and one enforced one, and the enforced one is the only one that has ever stopped anything.** Two directions, neither buildable from the catalog repo: (1) Gitea Actions CI — nothing exists to build on; (2) controller-side at template load (`internal/stacks/metadata.go:336-358`), but that needs a runtime probe on a customer box **and** a ruling on what a failing sync does to a customer's working app. Interim position recorded in the catalog's `CLAUDE.md`: run it when a template's `volumes:` or image tag changes, and at the start of every catalog campaign | operator |
|
||||
| **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC |
|
||||
| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | the operator's sizing decision | **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 | operator |
|
||||
| **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
|
||||
|
||||
## Why the TOP READY rows rank this way
|
||||
|
||||
|
||||
This covers the next few only — it is deliberately **not** a full ordering of the table above, so that
|
||||
there is one ranking to maintain rather than two.
|
||||
|
||||
@@ -105,3 +115,31 @@ there is one ranking to maintain rather than two.
|
||||
there is nothing for CC to build until it lands. Ranked here rather than omitted because it is the
|
||||
only item on this page about the *publish channel* of the most privileged artifact Felhom ships,
|
||||
and today's exposure is zero — which makes now the cheapest moment it will ever be to decide.
|
||||
|
||||
### The 2026-08-02 intake (R-156 … R-164), ranked
|
||||
|
||||
Filed in one pass from Campaign 10, its two spikes, and the 53-template catalog persistence sweep.
|
||||
**R-156 and R-157 had lived only in audit documents** — the identical *"minted in a spike doc and
|
||||
never carried across"* failure the register already records for R-153/R-154/R-155, caught by the
|
||||
sweep's own §8.0 while it was happening. R-158 was minted by a second session on the same day for an
|
||||
unrelated finding, which is why the sweep's proposals were renumbered to R-159…R-162 at filing time.
|
||||
|
||||
1. **R-157** — highest: a `deployed: true` app can stay **down indefinitely** after a power cut or
|
||||
hard reset, and in mechanism **B** nothing reports it on any channel (`0 currently down`). It is the
|
||||
only row here where the customer loses service and has no signal at all.
|
||||
2. **R-161** — next, because it is why R-156's class **recurs**. The gate that convicts it exists and
|
||||
nothing runs it; R-29's record is three orphaned gates and one enforced one, and only the enforced
|
||||
one has ever stopped anything. Needs a ruling, not a task.
|
||||
3. **R-156** — the class is now detectable and two of three apps are fixed; what remains is **papra's
|
||||
referral**, one app, well understood.
|
||||
4. **R-163** — a real ceiling that silently caps local backup once an app outgrows `mp1`, and it gates
|
||||
Tier-2 and Tier-3 as well. Ranked below the above only because **overflow itself is safe today** —
|
||||
it refuses per app and preserves the last good unit byte-identical. Waiting on the operator's ratio.
|
||||
5. **R-158** — the gap that makes R-163 dangerous: cross the size line and **one page** tells you. On
|
||||
its own it is a notification gap, not a silent failure, which is why it sits here and not higher.
|
||||
6. **R-164** — blocked on a predicate, no customer impact today; it only becomes urgent if the unit
|
||||
size in R-163 is judged unacceptable, since the tar-drop is the cheapest way to halve it.
|
||||
7. **R-162** — `WATCHING` only. A limitation that fails closed; revisit if a non-overlay driver ships.
|
||||
|
||||
**R-159 and R-160 are SHIPPED** and are not ranked; they are filed to record the class, and R-159's
|
||||
class (an image `VOLUME` at an unmounted path) is still live — `immich-server` has one today.
|
||||
|
||||
@@ -161,6 +161,15 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
|
||||
| R-128 | **`ISO_VERSION` "aligns with SCRIPT_VERSION" was a comment nothing evaluated** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | Closed by **correcting the claim, not asserting it**: the ISO is frozen while `felhom-host-install.sh` is fetched at run time from `main` (R-94/R-110), so an assertion would invent a constraint. `build-felhom-iso.sh:45-52`. Full reasoning in `OPEN-ITEMS.md` |
|
||||
| R-154 | **`[first-boot]` is automated-install-only, and nothing in the tree said so** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | A PVE property, measured with a same-image control (`audits/SPIKE-universal-iso-3-2026-07-31.md` §2); recorded at `scripts/iso/pkg/build-deb.sh:6-11`. Superseded in practice by the `.deb` delivery route |
|
||||
| R-155 | **`iso-repack.sh` refused any ISO without `auto-installer-mode.toml`** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | **Narrowed, not deleted** — unchanged for `FELHOM_MENU=single` (`iso-repack.sh:121-128`), does not apply to `release` where the file's absence *is* gate G1. Do not remove it wholesale |
|
||||
| R-156 | **An app's data is neither persisted nor backed up, and it reports healthy** — a template mounts a volume at a path the app never writes, so data sits in the container's writable layer: lost on redeploy, tarred nightly as an empty dir, healthcheck green | S | **DETECTED + GATE SHIPPED; papra REFERRED** (catalog sweep, 2026-08-02) | Found by Campaign 10 on **papra**; the 53-template sweep convicted **gramps-web** and **wishlist** too and **fixed both**. The gate is `app-catalog-felhom.eu/scripts/check-volume-persistence.py` — a runtime probe (`docker diff` + mount occupancy + writability, canary self-test, fails closed); the detector and the gate are one program. **papra is NOT fixed**: the fix needs the app to use `/app/data` or the template to mount `/app/app-data`, so it is referred. Sweep evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` (53 probe.json). Ranking in `OPEN-ITEMS.md` |
|
||||
| R-157 | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — two mechanisms** | M | READY — 2026-08-02 | R-52 built `internal/bootrecon` for exactly this (F5: two apps `Exited` ~18 h). **(A)** the sweep fires ~5 s after start (`main.go:236`) while docker is still restoring and never re-checks — **3 of 6 hard resets**, intermittent, which is what a race predicts and a wrong predicate does not; the predicate is sound (`bootrecon.go:100`; `IsDownState` includes `StateDegraded`, `manager.go:55`). **(B)** a mid-compose interruption leaves **zero containers**, the signature bootrecon deliberately skips as a user Stop — and deadapp then reported **`0 currently down`** for a `deployed: true` app that was not running. **A settle-condition fix closes A and leaves B open.** Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` |
|
||||
| R-158 | **A local Tier-1 backup failure reaches no hub channel** — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, only the off-box leg calls it | S | READY — 2026-08-02 | Fifth *seam built but never wired*; **R-97's defect one tier over**. Only caller `cmd/controller/main.go:659`; the manager's seams are tier2/offbox/offbox-enlarge (`backup.go:33,37,58`), none for the unit capture. **Ranked below R-157 by measurement:** `/backups` DOES show `✗ Adatmentés sikertelen`, it persists across a second failure and clears on recovery, and `/backups/apps` shows the last good unit's true mtime — so it is a notification gap, not silent failure. **The specific fix: `/backups/apps` is the page you would check for one app, and the only one that never says.** Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md`, `audits/CAMPAIGN-10-closeout-2026-08-02.md` |
|
||||
| R-159 | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy** | XS | **SHIPPED** (`templates/wishlist/`, 2026-08-02) — filed for the CLASS | Image declares `VOLUME /usr/src/app/data`; template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns names only for compose-declared volumes, so `DumpAppVolumes` never sees an anonymous one. **The class is open:** any image `VOLUME` at an unmounted path is silent unbacked-up storage — **`immich-server` has one today** at `/data`, empty when measured. Proposed `REUSE.md` rule: a template mounts every path in `Config.Volumes`, or says why not |
|
||||
| R-160 | **gramps-web persisted three paths and wrote to none of them** | XS | **SHIPPED** (`templates/gramps-web/`, 2026-08-02) | `/app/data` appears nowhere in the image's env; the accounts DB and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths, the template three, one a phantom. **Severity above papra's:** papra loses documents the customer may hold elsewhere; gramps-web loses the artefact built inside the app, of which no other copy exists by construction |
|
||||
| R-161 | **The volume-persistence gate exists and nothing automated runs it** | M | BLOCKED — needs a ruling (operator, 2026-08-02) | Enforcement is **convention**: the catalog repo has no CI at all (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none). **R-29's record: three orphaned gates, one enforced, and only the enforced one ever stopped anything.** Options, neither buildable from the catalog repo: Gitea Actions CI (nothing to build on), or controller-side at template load (`internal/stacks/metadata.go:336-358`) — which needs a runtime probe on a customer box and a ruling on what a failing sync does to a working app |
|
||||
| R-162 | **`docker diff` is the gate's only witness and its failure mode is quiet** | XS | WATCHING — 2026-08-02 | A limitation, not a defect. The gate's power is `docker diff` excluding mounted paths; on a driver where it is unsupported or lies, the gate degrades to mount-occupancy + writability **and would not say so**. It fails closed (the canary self-test stops reporting BROKEN and the gate then refuses to report), but the message blames the prober rather than the driver. Revisit only if a non-overlay driver ships |
|
||||
| R-163 | **`mp1` is RETENTION, not staging — and it is sized as if it were neither** | M | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | Settled by reading: a unit is the KEPT copy on the app's **own** drive (`backup.go:245-255`), falling back to the system SSD for driveless apps — *"the SSD-only system-data fallback"* (`appbackup/paths.go:26-27`); **no post-copy deletion**, the only prune is F5 residue-on-old-drives (`backup.go:1053-1112`). So `mp1` **20 G** retains every driveless app's unit while `mp0` permits **50 G** of volumes, and a DB app's unit is up to **~2×** its data (measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to 0 (`felhom-agent/.../main.go:178`), not derived from the drive; demo-hp's real guest ships 50G/20G. **`mp1` gates the whole chain** — Tier-2 mirrors the unit *"(always)"* (`tier2.go:302,368`) and Tier-3 carries it. Bounded: a unit is volume tars + DB dumps only, **never `mp8` userdata** (`recovery_unit.go:20-25`). **No number proposed.** → `07-backup-architecture.md` §7.5 |
|
||||
| R-164 | **C2's chain — the DB volume tar cannot be dropped until a sound dump predicate exists** | S | BLOCKED — on the predicate (2026-08-02) | The unit holds a volume tar **and** a SQL dump and the restore uses both: the dump is authoritative and replayed after the tar so it WINS (F17), with only the DB service up (R-47) — `restore_unit.go:262-266`. Dropping the DB tar would halve DB-app units and close R-127(b)'s initdb-skip trap. **The obvious gate is dead, measured:** `ValidateDump`'s empty-`accounts` warning was **correct** (the DB truly had 0 rows; seeding one stopped the warning and put the row in the dump) — but **a fresh appliance legitimately has zero accounts**, so gating on it blocks every new customer's first backup. Order: sound predicate (dump vs **live** per-table counts) → warn→gate → tar-drop. Pairs with **R-127** |
|
||||
| R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |
|
||||
| R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` |
|
||||
| R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect |
|
||||
|
||||
Reference in New Issue
Block a user