STATUS.md: a plain-language operator page, and today's four decisions recorded
Documentation only — no code, no box, no build.
STATUS.md (repo root, 652 words / 67 lines): what works · what's broken ·
what we're working on · waiting on you · changed since. A VIEW of
OPEN-ITEMS.md, holding nothing of its own; not CONTEXT.md, and both files
now say why they stay separate. No R-n is the subject of a sentence —
identifiers are bracketed pointers only.
CONTEXT.md S-5 records the four operator decisions taken 2026-08-02
(D-a … D-d), none of them implemented:
D-a merge mp1 into mp0 rather than resize it — before any external
install, and D-c ships in the same step → R-165
D-b desired/observed app state in its own store, with the state-store
safety rule verbatim → R-166 (BLOCKED)
D-c customer fill warning + operator backup-failure alert → R-167
D-d only DooPlex and Peti's box are protected → target-selection.md
R-163 RE-FRAMED, not closed: the sizing question is withdrawn rather than
answered; the row survives as the record of the constraint until R-165
lands. R-156's papra referral RESOLVED — deployed nowhere, so the template
fix strands nothing; the docker ps evidence is recorded with its
provenance and its scope limit.
target-selection.md: two protected machines, everything else disposable.
ep0 is no longer Tier 2 but is not scratch (it holds the only off-premises
copy of real customer data) — flagged for explicit operator confirmation.
The demo-box backup-target fence drops from prohibition to stated cost,
because D-d spends that reference anyway.
CLAUDE.md gains an End-of-session checklist carrying the STATUS.md
maintenance rule and "a finding goes in OPEN-ITEMS.md first".
This commit is contained in:
@@ -84,15 +84,18 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-137** | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with "<domain>"` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC |
|
||||
| **R-138** | **A shared-zone `cf_api_token` is a zone-wide DNS-write capability on a customer's box** — written 0600 to `/opt/docker/stacks/traefik/.env` (`controller/internal/infra/infra.go:123`) | READY (S) | — | Today each box holds a token for a zone nobody else uses, so the blast radius is one customer. Under a shared customer zone, one compromised tester box could repoint every other tester's DNS. The ACME path is already switchable — an empty token selects HTTP-01 (`traefik.yml.tmpl`) — so the fix is policy plus a guard that refuses to hand a shared-zone customer a zone-scoped token. Same audit §5.2 | CC |
|
||||
| **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC |
|
||||
| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. Next action is papra's referral only. See R-161 (nothing runs the gate) and R-159/R-160 | CC |
|
||||
| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. **THE REFERRAL IS RESOLVED, 2026-08-02 — papra is deployed NOWHERE, so the template fix strands nothing and can be applied.** The referral existed because changing where the volume mounts moves live data: an installed papra writes `db.sqlite` into the container's writable layer, and a remount relocates the path out from under it. With no instance deployed there is no live data to move, so the cheaper leg — **the template mounts `/app/app-data`** — is takeable directly, without waiting on upstream to adopt `/app/data`. **Provenance, stated because it decides the row:** the observation is `docker ps -a` on demo-hp's **guest 9201** returning empty, supplied with the 2026-08-02 task; **this session did not re-measure** (documentation-only, every box fenced). **Scope of that evidence, honestly:** it covers guest 9201 — the guest papra was convicted on in Campaign 10 — and **no other customer's guest was enumerated**, so a re-check belongs in the task that edits the template, before it edits it. **Next action: apply the template fix in `app-catalog-felhom.eu` and re-run `scripts/catalog_gates.py`** (deliberately not done here — that repo was out of scope for this task). See R-161 (nothing runs the gate automatically) and R-159/R-160 | CC |
|
||||
| **R-157** | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms, and a settle-condition fix closes only one.** **(A)** the sweep runs ~5 s after controller start (`cmd/controller/main.go:236`) while docker is still restoring, sees "no boot-orphaned apps", and never re-checks — **intermittent, 3 of 6 hard resets**; the predicate itself is sound (`bootrecon.go:100`, and `IsDownState` includes `StateDegraded`, `manager.go:55`). **(B)** an interruption that lands mid-compose leaves the stack with **zero containers**, which is exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp check reported **`0 currently down`** while a `deployed: true` app was not running: **silent on every channel**. | **READY (M)** | — | **Two fixes, not one.** A: a settle condition before the single observation, or one re-evaluation at the end of `deadAppBootGrace`, still bounded to the same attempts. B: needs a signature that distinguishes "customer stopped this" from "interrupted between containers" — B is F-CRIT-1's shape (`CLAUDE.md` false-invariant #4) reached via hard reset rather than quiesce. Test the **consequence** (app running again before the grace expires), not the mechanism. Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` | CC |
|
||||
| **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC |
|
||||
| **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `<project>_<name>` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC |
|
||||
| **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |
|
||||
| **R-161** | **The volume-persistence gate is enforced by CONVENTION, not automatically.** The catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none exists). | **REDUCED SCOPE — open** (operator ruling 2026-08-02) | a second person touching templates | **RULED. Both obvious enforcement points were rejected for measured reasons.** *Controller-side at template load:* rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — **it would pass on the exact defect it exists to catch**; the property is decidable only at runtime. *CI:* rejected for now — neither repo has any, and there are no users yet. **SHIPPED instead** (`app-catalog-felhom.eu` `fd7747d`): `scripts/catalog_gates.py`, ONE entry point running all three gates, non-zero exit on any failure, **mandated in the catalog's `CLAUDE.md`** the way `site_gates.py` is. Rationale for the record: of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md — `site_gates.py` is run, R-29's three orphans are named nowhere and have stopped nothing. **What remains open is only the automatic half:** this is convention, run by a person, and that is sufficient while one person touches templates. Revisit when a second does | operator |
|
||||
| **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC |
|
||||
| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | the operator's sizing decision | **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 | operator |
|
||||
| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio** | — (the sizing question is answered; the work is **R-165**) | **RE-FRAMED, NOT CLOSED (operator decision D-a, 2026-08-02 — `CONTEXT.md` S-5).** The row asked *what ratio should `mp1` be?* and that question is **withdrawn rather than answered**: `mp1` is merged into `mp0` so local recovery units share the app-data area and the ceiling stops existing — a bigger number is the same wall further away. **This row stays open as the record of the constraint** (what `mp1` is for, what it gates, and the measured 2× DB-app unit size) **until the merge lands**, because until then every consequence below is still live on every box. **The work is R-165; the warning that must ship with it is R-167.** Original finding, unchanged, follows. **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 (widened to R-167) | CC |
|
||||
| **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
|
||||
| **R-165** | **Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.** Operator decision **D-a**, 2026-08-02 (`CONTEXT.md` S-5). Local recovery units share the app-data area instead of holding their own fixed ceiling, so the wall R-163 describes is removed rather than moved further away. Guest 9201 on demo-hp ships `mp0 50G / mp1 20G` today | **READY (M) — NEW 2026-08-02** | — | **Two conditions travel WITH the decision and are not optional.** **(1) Before any external install.** It changes the **disk layout**, so it is a fresh-install shape while there are no external boxes and a per-box migration after — and the decision's cheapness is entirely a function of that ordering. **(2) It removes a wall that currently fails safely**, so **R-167** (D-c: fill warning + failure alert) lands in the same step, never after: today an app that outgrows `mp1` is refused per app with the last good unit preserved byte-identical (R-158's measurement), and after the merge the same overflow consumes the space the app itself is using. Touches the installer/agent guest shape (`--sysdata-grow` defaults to **0** and is not derived from the physical drive, `felhom-agent/cmd/felhom-agent/main.go:178`) and the golden. **Does NOT close R-163** — that row is the record of the constraint and stays open until this lands | CC |
|
||||
| **R-166** | **App state gets a desired/observed model with its own store.** Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5). Three kinds of information, deliberately separated: **desired** (running, or stopped because the customer said so) stays in `app.yaml`; **in-flight operations** (a backup started and never reported finishing) are **persisted** — written before the operation and after it — so an interruption is visible after a reboot; **observed** (unhealthy, restarting, drive gone) is **not** persisted and is rebuilt by looking. The last two live in a store **separate from `app.yaml`**, so frequent writes never touch a file holding deploy settings and secrets | **BLOCKED — NEW 2026-08-02** | two facts to establish before it is specced | **NOT a spec yet, deliberately.** **Establish first:** (a) whether the crash-safe journal already in the backup code covers the in-flight case — *it may already exist and merely be unwired, which has been the answer four times in this project* (`CLAUDE.md` "seam built but never wired"); (b) whether the existing SQLite store is reachable from this code path. **Binding safety rule, verbatim from the decision:** *losing the state store must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the worst acceptable outcome is re-running a backup that already ran.* The controller must also check **every container of an app**, so a partly-dead app cannot read as healthy. **This is the durable fix for R-157 mechanism B** (zero containers reading as a deliberate stop) and for F-CRIT-1's class; R-157 keeps its own row because it wants a bounded fix now, not this rework | CC |
|
||||
| **R-167** | **Storage monitoring and backup alerts.** Operator decision **D-c**, 2026-08-02 (`CONTEXT.md` S-5). Two pieces: a warning to the **customer** as a drive fills, **before** anything fails; and a failure alert to the **hub/operator** when any backup fails for **any** reason, lack of space included | **READY (M) — NEW 2026-08-02** | — | **Prerequisite for R-165** — D-a removes a wall that currently fails safely, so this ships in the same step, not after. **Subsumes and widens R-158**, which is the same alert one tier down (`NotifyBackupFailed` exists and the hub allowlists `backup_failed`; only the off-box/NAS leg calls it, `cmd/controller/main.go:659`). R-158 stays as the named seam and its proposed `unitNotify` shape is the concrete first leg — **do not file a second row for the same wire.** The customer half is new: nothing today warns before a fill, and R-158's measurement is that when it happens exactly one page says so | CC |
|
||||
|
||||
## Why the TOP READY rows rank this way
|
||||
|
||||
@@ -132,7 +135,10 @@ unrelated finding, which is why the sweep's proposals were renumbered to R-159
|
||||
ran the gate; it now has a mandated entry point, so R-156's residue is the larger remaining item.)*
|
||||
4. **R-163** — a real ceiling that silently caps local backup once an app outgrows `mp1`, and it gates
|
||||
Tier-2 and Tier-3 as well. Ranked below the above only because **overflow itself is safe today** —
|
||||
it refuses per app and preserves the last good unit byte-identical. Waiting on the operator's ratio.
|
||||
it refuses per app and preserves the last good unit byte-identical. **RE-FRAMED 2026-08-02:** no
|
||||
longer waiting on a ratio — decision **D-a** merges `mp1` away, so the row is now the record of the
|
||||
constraint and the work moves to **R-165** (with **R-167** shipping in the same step). R-165 inherits
|
||||
this rank; it is the highest-ranked item that must land **before any external install**.
|
||||
5. **R-158** — the gap that makes R-163 dangerous: cross the size line and **one page** tells you. On
|
||||
its own it is a notification gap, not a silent failure, which is why it sits here and not higher.
|
||||
6. **R-164** — blocked on a predicate, no customer impact today; it only becomes urgent if the unit
|
||||
|
||||
@@ -2,28 +2,48 @@
|
||||
|
||||
> Read before picking a machine for a drill, a destructive test, or a throwaway VM. It answers one
|
||||
> question: *what is safe to lose.* Reachability is in `CLAUDE.md`; hardware in `operations/nodes.md`.
|
||||
> Added 2026-07-30.
|
||||
> Added 2026-07-30. **Revised 2026-08-02 by operator decision D-d** (`CONTEXT.md` S-5).
|
||||
|
||||
## The rule
|
||||
|
||||
> **Start at Tier 0 and work down only if Tier 0 genuinely cannot host the work.**
|
||||
> **A Tier 2 machine is used only when a task says so explicitly** — never by inference from what was
|
||||
> **Two machines are protected: `DooPlex` and Peti's box. Everything else is disposable.**
|
||||
> Operator decision **D-d**, 2026-08-02. DooPlex because it holds Gitea, the hub, the backups and the
|
||||
> registry — everything else rebuilds from it, and it rebuilds from nothing. Peti's box because there
|
||||
> is a real person behind it. **Every other box, both demo boxes included, may be broken or
|
||||
> reinstalled freely.**
|
||||
>
|
||||
> **This is a correction, not a relaxation.** The earlier posture was costing whole sessions to
|
||||
> caution and pushing drills onto DooPlex — the one machine that should never host them. If you are
|
||||
> weighing whether a demo box can take a destructive test: it can. **Reach for a Tier 0 box first and
|
||||
> do not ask.**
|
||||
>
|
||||
> Still true: **start at Tier 0 and work down only if Tier 0 genuinely cannot host the work**, and a
|
||||
> protected machine is used only when a task says so explicitly — never by inference from what was
|
||||
> not forbidden. An absent fence is not permission. If no tier fits, **stop and ask.**
|
||||
|
||||
Fences name **acts**, not machines. "Do not re-target demo-hp's backup target" and "do not use demo-hp
|
||||
to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine
|
||||
Fences name **acts**, not machines. "Do not destroy demo-hp's `drill-r50` fixture" and "do not use
|
||||
demo-hp to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine
|
||||
prohibition as covering the act it names and nothing more.
|
||||
|
||||
| Tier | Meaning | Machines |
|
||||
|---|---|---|
|
||||
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
|
||||
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
|
||||
| **2 — precious. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **ep0** + the Hetzner Storage Boxes |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
|
||||
|
||||
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
|
||||
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
|
||||
thing under test, the source of truth for it, and the backups, at once.
|
||||
|
||||
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
|
||||
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
|
||||
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
|
||||
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
|
||||
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
|
||||
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
|
||||
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
|
||||
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
|
||||
|
||||
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
|
||||
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
|
||||
|
||||
@@ -38,20 +58,24 @@ felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat
|
||||
**`/mnt/nvme-1tb`, at its root** (a subdirectory fails the agent's `exactMount` check → storage reads
|
||||
`disconnected` forever). **That warning is about one storage, not the box.** `/mnt/nvme-1tb` is also
|
||||
the `felhom-backup` target and the enrolled user-data drive, so remove scratch storages when done.
|
||||
- **Forbidden:** do not re-target its backup configuration (below). Do not destroy or unblock
|
||||
**`drill-r50` (VM 300)** — the only drift fixture (R-93). No baked SSH key; access is the hub-vaulted
|
||||
G1 break-glass password (R-61).
|
||||
- **Forbidden:** do not destroy or unblock **`drill-r50` (VM 300)** — the only drift fixture (R-93).
|
||||
(Access: the docs say no baked SSH key and G1 break-glass, but a key authenticated on 2026-07-31 —
|
||||
**R-129**, unresolved.)
|
||||
|
||||
### `demo-felhom` — N100 · **Tier 0**
|
||||
|
||||
- **Freely:** create/destroy guests and scratch customers; reinstall the box.
|
||||
- **Care:** it carries the **PBS-DR / offsite tier** (demo-hp has none), so it is the Tier 0 box whose
|
||||
backup chain a test can actually disturb. Prefer demo-hp, per the 2026-07-25 ruling.
|
||||
- **Forbidden:** do not re-target its backup configuration.
|
||||
|
||||
**Both Tier 0 boxes, one shared prohibition and its reason:** do not re-point either backup target.
|
||||
They are the only two **correctly configured** boxes, so they are the regression path new installer
|
||||
logic is measured against — losing that reference costs more than the box does.
|
||||
**Both Tier 0 boxes — the shared backup-target fence is DOWNGRADED to a cost, 2026-08-02 (D-d).** It
|
||||
read *"do not re-point either backup target"*, because these are the only two **correctly configured**
|
||||
boxes and therefore the regression path new installer logic is measured against. D-d makes both boxes
|
||||
freely breakable and reinstallable, which loses that reference just as thoroughly — so the fence was
|
||||
inconsistent with the decision and is not kept as a prohibition. **What survives is the reason:**
|
||||
re-pointing (or reinstalling) costs the reference configuration, so know that you are spending it and
|
||||
put the box back. If both are spent at once there is no correctly-configured box left to compare
|
||||
against.
|
||||
|
||||
### `DooPlex` — 192.168.0.180 · **Tier 2**
|
||||
|
||||
@@ -71,7 +95,7 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
|
||||
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
|
||||
enrolled host. No access route from DooPlex, and nothing here needs one.
|
||||
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2**
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
|
||||
|
||||
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
||||
path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
|
||||
|
||||
Reference in New Issue
Block a user