register: file R-156..R-164 in one pass, ranked; and record what mp1 is actually for
Nine rows into OPEN-ITEMS.md and ROADMAP.md, matching each file's column shape. R-156 and R-157 had lived only in audit documents - the identical "minted in a spike doc and never carried across" failure the register already records for R-153/R-154/R-155, caught by the catalog sweep's own section 8.0 while it was happening. R-158 was minted by a second session the same day for an unrelated finding, which is why the sweep's proposals were renumbered R-159..R-162 at filing time. All nine IDs verified free in BOTH backlog files before use. Part 0 settled the question the sizing item depended on, by reading: mp1 is RETENTION, not staging, and neither of the two framings was right. A unit is the KEPT copy on the app's OWN drive (backup.go:245-255); for an app with no HDD_PATH the namespace falls back to the system SSD - "the SSD-only system-data fallback" (appbackup/paths.go:26-27). There is no post-copy deletion: the only prune is F5 residue-on-old-drives when an app MOVES (backup.go:1053-1112). So mp1 retains the units of driveless apps only - not every app, but not transient either. Confirmed against the spike: sys_drive held exactly the four driveless apps and not calibre-web, which had a drive and was still backed up. A unit is volume tars + DB dumps only, never mp8 userdata (recovery_unit.go:20-25), so a 1 TB photo library can never overflow one. And mp1 gates the WHOLE chain, not just Tier 1: Tier-2 mirrors the unit "(always)" from RecoveryUnitPath (tier2.go:302,368) and Tier-3 carries it, so a unit that cannot be written leaves both with nothing to copy. Part 2 fired on both triggers - retention, and the fallback undocumented - so 07-backup-architecture.md gains section 7.5. Section 6.1 said a unit lives "on the app's own drive", which is true and was the whole story only for drive-resident apps; the no-drive case was undocumented, as was the sizing constraint. 7.5 records the mp0-50G-vs-mp1-20G mismatch, the measured ratios (DB app up to ~2x, 21.1GB -> 40.2GB; file-only 1.00x), and the bound this puts on D5's Lane-1 independence: restorable from the drive alone only while the unit still fits - about 19 GB file-only, about 10 GB DB-backed. No number proposed; the ratio is the operator's ruling (R-163). R-159/R-160 marked SHIPPED only after verifying the template changes are in app-catalog origin/main, and R-156's gate likewise (check-volume-persistence.py present). papra is NOT fixed - referred - so R-156 stays open on that one app. Ranked, with one line of reasoning each: R-157 first (an app can stay down indefinitely with mechanism B silent on every channel), then R-161 (the gate exists and nothing runs it, which is why R-156's class recurs - R-29's record is three orphaned gates and one enforced), R-156, R-163, R-158, R-164, R-162.
This commit is contained in:
@@ -84,9 +84,19 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-137** | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with "<domain>"` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC |
|
||||
| **R-138** | **A shared-zone `cf_api_token` is a zone-wide DNS-write capability on a customer's box** — written 0600 to `/opt/docker/stacks/traefik/.env` (`controller/internal/infra/infra.go:123`) | READY (S) | — | Today each box holds a token for a zone nobody else uses, so the blast radius is one customer. Under a shared customer zone, one compromised tester box could repoint every other tester's DNS. The ACME path is already switchable — an empty token selects HTTP-01 (`traefik.yml.tmpl`) — so the fix is policy plus a guard that refuses to hand a shared-zone customer a zone-scoped token. Same audit §5.2 | CC |
|
||||
| **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC |
|
||||
| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. Next action is papra's referral only. See R-161 (nothing runs the gate) and R-159/R-160 | CC |
|
||||
| **R-157** | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms, and a settle-condition fix closes only one.** **(A)** the sweep runs ~5 s after controller start (`cmd/controller/main.go:236`) while docker is still restoring, sees "no boot-orphaned apps", and never re-checks — **intermittent, 3 of 6 hard resets**; the predicate itself is sound (`bootrecon.go:100`, and `IsDownState` includes `StateDegraded`, `manager.go:55`). **(B)** an interruption that lands mid-compose leaves the stack with **zero containers**, which is exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp check reported **`0 currently down`** while a `deployed: true` app was not running: **silent on every channel**. | **READY (M)** | — | **Two fixes, not one.** A: a settle condition before the single observation, or one re-evaluation at the end of `deadAppBootGrace`, still bounded to the same attempts. B: needs a signature that distinguishes "customer stopped this" from "interrupted between containers" — B is F-CRIT-1's shape (`CLAUDE.md` false-invariant #4) reached via hard reset rather than quiesce. Test the **consequence** (app running again before the grace expires), not the mechanism. Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` | CC |
|
||||
| **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC |
|
||||
| **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `<project>_<name>` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC |
|
||||
| **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |
|
||||
| **R-161** | **The volume-persistence gate exists and nothing automated runs it.** `check-volume-persistence.py` is enforced by **convention**: the catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — none checked, none exists). | **BLOCKED** — needs a decision, not a task | a runner, or a cross-repo design ruling | This project's own record says convention is not enough: **R-29 has three orphaned gates and one enforced one, and the enforced one is the only one that has ever stopped anything.** Two directions, neither buildable from the catalog repo: (1) Gitea Actions CI — nothing exists to build on; (2) controller-side at template load (`internal/stacks/metadata.go:336-358`), but that needs a runtime probe on a customer box **and** a ruling on what a failing sync does to a customer's working app. Interim position recorded in the catalog's `CLAUDE.md`: run it when a template's `volumes:` or image tag changes, and at the start of every catalog campaign | operator |
|
||||
| **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC |
|
||||
| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | the operator's sizing decision | **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 | operator |
|
||||
| **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
|
||||
|
||||
## Why the TOP READY rows rank this way
|
||||
|
||||
|
||||
This covers the next few only — it is deliberately **not** a full ordering of the table above, so that
|
||||
there is one ranking to maintain rather than two.
|
||||
|
||||
@@ -105,3 +115,31 @@ there is one ranking to maintain rather than two.
|
||||
there is nothing for CC to build until it lands. Ranked here rather than omitted because it is the
|
||||
only item on this page about the *publish channel* of the most privileged artifact Felhom ships,
|
||||
and today's exposure is zero — which makes now the cheapest moment it will ever be to decide.
|
||||
|
||||
### The 2026-08-02 intake (R-156 … R-164), ranked
|
||||
|
||||
Filed in one pass from Campaign 10, its two spikes, and the 53-template catalog persistence sweep.
|
||||
**R-156 and R-157 had lived only in audit documents** — the identical *"minted in a spike doc and
|
||||
never carried across"* failure the register already records for R-153/R-154/R-155, caught by the
|
||||
sweep's own §8.0 while it was happening. R-158 was minted by a second session on the same day for an
|
||||
unrelated finding, which is why the sweep's proposals were renumbered to R-159…R-162 at filing time.
|
||||
|
||||
1. **R-157** — highest: a `deployed: true` app can stay **down indefinitely** after a power cut or
|
||||
hard reset, and in mechanism **B** nothing reports it on any channel (`0 currently down`). It is the
|
||||
only row here where the customer loses service and has no signal at all.
|
||||
2. **R-161** — next, because it is why R-156's class **recurs**. The gate that convicts it exists and
|
||||
nothing runs it; R-29's record is three orphaned gates and one enforced one, and only the enforced
|
||||
one has ever stopped anything. Needs a ruling, not a task.
|
||||
3. **R-156** — the class is now detectable and two of three apps are fixed; what remains is **papra's
|
||||
referral**, one app, well understood.
|
||||
4. **R-163** — a real ceiling that silently caps local backup once an app outgrows `mp1`, and it gates
|
||||
Tier-2 and Tier-3 as well. Ranked below the above only because **overflow itself is safe today** —
|
||||
it refuses per app and preserves the last good unit byte-identical. Waiting on the operator's ratio.
|
||||
5. **R-158** — the gap that makes R-163 dangerous: cross the size line and **one page** tells you. On
|
||||
its own it is a notification gap, not a silent failure, which is why it sits here and not higher.
|
||||
6. **R-164** — blocked on a predicate, no customer impact today; it only becomes urgent if the unit
|
||||
size in R-163 is judged unacceptable, since the tar-drop is the cheapest way to halve it.
|
||||
7. **R-162** — `WATCHING` only. A limitation that fails closed; revisit if a non-overlay driver ships.
|
||||
|
||||
**R-159 and R-160 are SHIPPED** and are not ranked; they are filed to record the class, and R-159's
|
||||
class (an image `VOLUME` at an unmounted path) is still live — `immich-server` has one today.
|
||||
|
||||
Reference in New Issue
Block a user