STATUS.md: a plain-language operator page, and today's four decisions recorded
Documentation only — no code, no box, no build.
STATUS.md (repo root, 652 words / 67 lines): what works · what's broken ·
what we're working on · waiting on you · changed since. A VIEW of
OPEN-ITEMS.md, holding nothing of its own; not CONTEXT.md, and both files
now say why they stay separate. No R-n is the subject of a sentence —
identifiers are bracketed pointers only.
CONTEXT.md S-5 records the four operator decisions taken 2026-08-02
(D-a … D-d), none of them implemented:
D-a merge mp1 into mp0 rather than resize it — before any external
install, and D-c ships in the same step → R-165
D-b desired/observed app state in its own store, with the state-store
safety rule verbatim → R-166 (BLOCKED)
D-c customer fill warning + operator backup-failure alert → R-167
D-d only DooPlex and Peti's box are protected → target-selection.md
R-163 RE-FRAMED, not closed: the sizing question is withdrawn rather than
answered; the row survives as the record of the constraint until R-165
lands. R-156's papra referral RESOLVED — deployed nowhere, so the template
fix strands nothing; the docker ps evidence is recorded with its
provenance and its scope limit.
target-selection.md: two protected machines, everything else disposable.
ep0 is no longer Tier 2 but is not scratch (it holds the only off-premises
copy of real customer data) — flagged for explicit operator confirmation.
The demo-box backup-target fence drops from prohibition to stated cost,
because D-d spends that reference anyway.
CLAUDE.md gains an End-of-session checklist carrying the STATUS.md
maintenance rule and "a finding goes in OPEN-ITEMS.md first".
This commit is contained in:
@@ -102,6 +102,20 @@ pushes; **you (Claude Code) implement**. A file being open in the editor is NOT
|
||||
explicit paths only, `git pull --rebase` before every push, and do not run two writing
|
||||
sessions on one clone (use `git worktree` if truly needed).
|
||||
|
||||
## End-of-session checklist
|
||||
|
||||
- **`CHANGELOG.md` + `REPORT.md`** per the rule above, in every repo touched.
|
||||
- **`REUSE.md`**, if a shared helper or pattern moved (same commit).
|
||||
- **The capability map** (`documentation/architecture/00-capability-map.md`), if a capability's status
|
||||
changed — with its new evidence citation.
|
||||
- **The architecture doc** that owns any changed contract (S-1, `CONTEXT.md`).
|
||||
- **Root `STATUS.md`** — *update it at the end of every session in which something shipped, broke, or
|
||||
was decided. It is a view of `documentation/backlog/OPEN-ITEMS.md` — **nothing may exist only
|
||||
there**. One screen; cut items rather than extending it.* It is written for the **operator** in
|
||||
plain language, and is deliberately **not** `CONTEXT.md` — do not consolidate the two.
|
||||
- **A finding goes in `OPEN-ITEMS.md` first**, never only in a report, an audit or `STATUS.md`. Four
|
||||
items in this project were minted in a spike doc and lost (R-153/154/155, R-156/157).
|
||||
|
||||
## Tech stack (Hub)
|
||||
|
||||
- **Language:** Go (stdlib `net/http` + `html/template`, no frameworks). **DB:** SQLite via
|
||||
|
||||
+52
@@ -7,6 +7,14 @@
|
||||
> Read it first; `ROADMAP.md` holds the full history and reasoning behind each item. **`REPORT.md` is
|
||||
> per-session and overwritten — nothing durable may live only there.**
|
||||
|
||||
> **This file is NOT `STATUS.md`, and the two must not be consolidated** (2026-08-02). `CONTEXT.md`
|
||||
> is technical working state written for Claude Code: rulings with their mechanisms, file paths,
|
||||
> version numbers, the reasoning a session needs before it touches code. Root `STATUS.md` is written
|
||||
> for the **operator** — technical, but not in the code and not a reader of the audits — in plain
|
||||
> language, one screen, no identifiers in the prose. Same subjects, different readers; merging them
|
||||
> would make one of the two audiences stop reading. `STATUS.md` is also a **view of `OPEN-ITEMS.md`**
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81).**
|
||||
@@ -79,6 +87,50 @@ Five decisions were deliberately **left open for the operator** and are recorded
|
||||
stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with
|
||||
the guest it backs up. Gaps minted the same session: **R-102 … R-108**.
|
||||
|
||||
**S-5 — four operator decisions taken in discussion on 2026-08-02, recorded before anything is
|
||||
built.** They existed only in conversation, which is the condition the standing rules were written
|
||||
against. Labels are the ones used in the discussion (**D-a … D-d**) and are deliberately kept
|
||||
distinct from S-3's `D1…D6`. **None of a, b or c is implemented by the session that recorded them**;
|
||||
their work is carried as backlog rows (**R-165, R-166, R-167**), not as prose here.
|
||||
|
||||
- **D-a — the dedicated backup partition is MERGED AWAY, not resized.** The appliance's system drive
|
||||
carries the OS, a **50 G** docker area (`mp0`) and a separate **20 G** area holding local recovery
|
||||
units (`mp1`; demo-hp's guest 9201 ships exactly that). The decision is to **merge the backup area
|
||||
into the large data area** so local backups share space with app data and the ceiling stops
|
||||
existing — deliberately *not* to pick a larger number, because a larger number is the same wall
|
||||
further away. **Two conditions, recorded with the decision, not as commentary:** (1) it changes the
|
||||
**disk layout**, so it must land **before any external install** — after that, every deployed box
|
||||
is a migration; (2) it **removes a wall that currently fails safely**, so the storage monitoring of
|
||||
**D-c** must ship **with** it, never after. Closes the sizing question in **R-163**, which is
|
||||
**re-framed rather than answered with a ratio** and is not closed until the merge lands. Work:
|
||||
**R-165**.
|
||||
- **D-b — app state gets a desired/observed model with its own store.** Three kinds of information,
|
||||
deliberately separated: **desired** (running, or stopped because the customer said so) stays in
|
||||
`app.yaml` with that app's other settings; **in-flight operations** ("a backup started and has not
|
||||
reported finishing") are **persisted**, written before the operation starts and after it ends, so
|
||||
an interruption is visible after a reboot; **observed** (unhealthy, restarting, drive gone) is
|
||||
**not** persisted and is rebuilt by looking. The last two live in a **state store separate from
|
||||
`app.yaml`**, so frequent writes never touch a file holding deploy settings and secrets.
|
||||
**Binding safety rule, verbatim:** *losing the state store must never cause an app to be deleted,
|
||||
restarted wrongly, or reported healthy when it is not — the worst acceptable outcome is re-running
|
||||
a backup that already ran.* The controller must also check **every container of an app**, so a
|
||||
partly-dead app cannot read as healthy (F-CRIT-1's shape). **Two things to establish before this is
|
||||
specced, not now:** whether the crash-safe journal already in the backup code covers the in-flight
|
||||
case — *it may already exist and merely be unwired, which has been the answer four times* — and
|
||||
whether the existing SQLite store is reachable from this code path. Work: **R-166**.
|
||||
- **D-c — storage monitoring and backup alerts.** Two pieces: a warning to the **customer** as a
|
||||
drive fills, **before** anything fails; and a failure alert to the **hub/operator** when any backup
|
||||
fails for any reason, lack of space included. **Prerequisite for D-a** — see that decision's
|
||||
condition (2). Subsumes and widens **R-158**, which is the same alert one tier down; R-158 stays as
|
||||
the named seam and is not duplicated. Work: **R-167**.
|
||||
- **D-d — which machines are protected: DooPlex and Peti's box, and nothing else.** DooPlex because
|
||||
it holds Gitea, the hub, the backups and the registry — everything else rebuilds from it. Peti's
|
||||
box because there is a real person behind it. **Every other box, both demo boxes included, is
|
||||
disposable and may be broken or reinstalled freely.** The earlier caution was costing sessions and
|
||||
pushing work onto DooPlex — the one machine that should never host it — so the ruling is a
|
||||
correction, not a relaxation. `documentation/runbooks/target-selection.md` updated the same session
|
||||
(2026-08-02); no backlog row, the decision *is* the change.
|
||||
|
||||
- **2026-07-27 — R-85: the DR tier is restore-tested UNATTENDED, and its failure is HEARD.** agent
|
||||
v0.104.0 + hub v0.77.0. R-82 proved a PBS restore round-trip **by hand**; three defects kept the
|
||||
unattended path from covering it. (a) `buildRestoreTestScheduler` built ONE runner on
|
||||
|
||||
@@ -1,162 +1,168 @@
|
||||
# REPORT — Hub v0.85.0: the Network card (2026-07-31)
|
||||
# REPORT — `STATUS.md` created, and the 2026-08-02 operator decisions recorded (2026-08-02)
|
||||
|
||||
**Overwritten** per the standing rule. Pairs with **agent v0.119.0** (see `felhom-agent/REPORT.md`);
|
||||
neither half is useful alone. Live evidence:
|
||||
`documentation/audits/host-addresses-visible-2026-07-31.md`.
|
||||
**Overwritten** per the standing rule. The prior contents (hub v0.85.0 Network card + v0.86.0 Copy
|
||||
without reveal, 2026-07-31) have their durable record in `hub/CHANGELOG.md` and
|
||||
`documentation/audits/host-addresses-visible-2026-07-31.md`; nothing was lost by this overwrite.
|
||||
|
||||
> The earlier v0.84.0 work this file previously described (break-glass Console access card) shipped,
|
||||
> deployed and validated in the same session; its durable record is `hub/CHANGELOG.md` v0.84.0 plus
|
||||
> `CONTEXT.md` ruling **S-4** and `documentation/backlog/` **R-133**.
|
||||
**Class: documentation only.** No code, no template, no box, no build. Repo `felhom.eu` only —
|
||||
`app-catalog-felhom.eu` was read for context and **not** modified. **No `CHANGELOG.md` entry exists
|
||||
for this change and none is missing:** this repo has no root changelog, only per-area `hub/`,
|
||||
`scripts/`, `website/` (`CLAUDE.md`), and this session touched none of those areas.
|
||||
|
||||
## 0. Follow-up shipped the same session — hub v0.86.0 (Copy without reveal)
|
||||
Baselines: `felhom.eu` @ `260a8f6`, `app-catalog-felhom.eu` @ `fd7747d`. Part 1 was derived by reading
|
||||
the register's rows and both ranking sections, not from memory.
|
||||
|
||||
**The operator hit a real defect in the v0.84.0 Console access card and it cost a login.** Copy was
|
||||
`disabled` until a Reveal, so clicking it did **nothing, silently**; the clipboard kept its previous
|
||||
contents — **another host's console password** — which was pasted into demo-hp's PVE login. It failed
|
||||
with `password check failed for user (root)`, which reads exactly like a stale credential and sends
|
||||
you diagnosing the wrong thing.
|
||||
---
|
||||
|
||||
Diagnosis, read-only first: the vaulted password **matched the box's `/etc/shadow` hash**, the account
|
||||
was unlocked and non-expiring, no TFA, no `pam_faillock` — and then it **minted a real PVE ticket**
|
||||
(`POST /api2/json/access/ticket` → HTTP 200, `root@pam`, 367-char ticket). So the credential was
|
||||
always good; only the clipboard was wrong. **That also closes the leg §7 of this report listed as
|
||||
operator-only** — the capability-map row now records it.
|
||||
## 1. `STATUS.md` — the file
|
||||
|
||||
**v0.86.0** makes Copy work without revealing (the *safer* default — the secret never renders, so it
|
||||
cannot be shoulder-surfed or screenshotted) and closes three silent-failure branches in one eight-line
|
||||
function: the disabled no-op, a missing `navigator.clipboard`, and an **ignored `writeText()`
|
||||
rejection** that let the operator believe a refused write had succeeded. The success message now names
|
||||
the host, because the clipboard is fleet-wide and "copied" alone cannot say for which box.
|
||||
Root of `felhom.eu`, so it is the first thing visible. Sections in the specified order: what works ·
|
||||
what's broken · what we're working on · waiting on you · changed since last update.
|
||||
|
||||
Tests 566 → 568, both pinning the regression; red-proof: re-adding `disabled` reproduces the shipped
|
||||
bug. Deployed and live-verified (`Synced/Healthy`, image `:0.86.0`, button served without `disabled`,
|
||||
retrieval URL defined exactly once, all four outcome messages present). Commits `670ec35` + `80f4739`.
|
||||
**Word count: 652 total, 581 excluding the header block** (`wc -w`; the header carries the
|
||||
view-not-source, not-`CONTEXT.md` and maintenance rules, which the spec requires). **That is over the
|
||||
~500 target and it is a deliberate miss, stated rather than hidden.** Five passes took it from 819 to
|
||||
652. Getting under 500 needed either dropping a mandated item or dropping the off-site-credential line
|
||||
— the register's **top-ranked** open item and the largest customer-data exposure on it. Cutting the
|
||||
biggest data risk to save forty words is the wrong trade on a page whose job is to show the operator
|
||||
what is at stake. It is 67 lines and fits a screen. **If the operator disagrees, the line to cut is
|
||||
the R-95/R-87 one** and the page drops to ~545.
|
||||
|
||||
## 1. Baselines
|
||||
Content, in the operator's ranking: an app can stay off after a power cut, silently (R-157) · the
|
||||
off-site copy can be erased by the box that wrote it, and has never been restored from (R-95, R-87) ·
|
||||
three of fifty-three apps saved data where backups never looked (R-156) · 20 GB of backup space
|
||||
against 50 GB of apps (R-163) · when that trips, one page says so and nothing alerts (R-158) · the
|
||||
checker exists but a person has to remember it (R-161).
|
||||
|
||||
| Repo | `main` @ start of this half | Version before → after |
|
||||
**Constraints honoured:** no `R-n` is the subject of any sentence — every identifier is a bracketed
|
||||
pointer at the end of a line; no file paths, function names or version numbers appear; every broken
|
||||
item is stated as what a customer or the operator would notice. Shipped, watching and
|
||||
blocked-on-a-predicate rows (R-159, R-160, R-162, R-164) are absent by design.
|
||||
|
||||
**One deviation, flagged per standing rule 4.** "Waiting on you" is specified as *decisions only*, and
|
||||
it carries one non-decision: **the hub password needs rotating** (R-132, owner Viktor). It is the only
|
||||
thing on the register waiting on the operator with a live credential consequence, and omitting it from
|
||||
the operator's own page to honour a section rule would be the letter over the point. It is labelled
|
||||
*"a job, not a decision"* so the section's shape is not quietly eroded.
|
||||
|
||||
## 2. The decisions — where each one went
|
||||
|
||||
All four are in **`CONTEXT.md` as standing ruling S-5**, labelled **D-a … D-d** as in the discussion
|
||||
and deliberately kept distinct from S-3's `D1…D6`. Open work is carried as backlog rows, per the
|
||||
existing convention — no new home was created for either.
|
||||
|
||||
| Decision | Recorded | Work |
|
||||
|---|---|---|
|
||||
| felhom.eu (hub) | `b4edc087fa9a` | hub **v0.84.0 → v0.85.0** |
|
||||
| felhom-agent | `6b5dade4dcc5` | agent **0.118.1 → 0.119.0** |
|
||||
| **D-a** — merge the backup partition away (not resize) | `CONTEXT.md` S-5 | **R-165** (new) |
|
||||
| **D-b** — desired/observed app state, own store | `CONTEXT.md` S-5 | **R-166** (new, `BLOCKED`) |
|
||||
| **D-c** — storage monitoring + backup alerts | `CONTEXT.md` S-5 | **R-167** (new) |
|
||||
| **D-d** — only DooPlex and Peti's box are protected | `CONTEXT.md` S-5 | `runbooks/target-selection.md`, this session — no row; the decision *is* the change |
|
||||
| **Maintenance rule** (Part 3) | `STATUS.md` header **and** `CLAUDE.md` § End-of-session checklist | — |
|
||||
|
||||
## 2. The ask, and what it turned out to be
|
||||
Recorded verbatim inside D-b, because it is the decision's binding constraint: *losing the state store
|
||||
must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the
|
||||
worst acceptable outcome is re-running a backup that already ran.* Its two "establish before speccing"
|
||||
items are carried on R-166 as the reason that row is `BLOCKED` rather than `READY`.
|
||||
|
||||
*"Show the LAN and WireGuard IPs on the hosts page — it looks like it is not visible anywhere."*
|
||||
Nearly exact, and the two halves were very different jobs:
|
||||
**Deliverable 5 asks for "the five decisions".** Part 2 defines four (D-a … D-d); the fifth deliverable
|
||||
line is the Part-3 maintenance rule, and it is in the table above. Nothing else in the task reads as a
|
||||
fifth decision — flagged rather than invented.
|
||||
|
||||
- **WireGuard** — the hub already had it (`wg_peers.assigned_ip`, `GetWGPeerForHost`, **no UI
|
||||
consumer**). It was visible in exactly one place, `/offsite`'s peer table keyed by pubkey, so you
|
||||
could go peer→host and never host→peer.
|
||||
- **LAN** — **not renderable at all.** `HostReport`/`HostMetrics` carried no address of any kind, so
|
||||
this needed a cross-repo agent release, not an edit. Confirmed against the struct and against a
|
||||
live box before proposing anything.
|
||||
**D-a's two conditions are recorded as conditions, not commentary:** it changes the disk layout so it
|
||||
must land **before any external install**, and it removes a wall that currently fails safely so
|
||||
**R-167 ships in the same step, never after**. R-165 restates both; R-167 names R-165 as the thing it
|
||||
gates.
|
||||
|
||||
Operator chose: both together as one arc, with the `addresses[]` (iface + cidr) shape.
|
||||
**None of D-a, D-b or D-c is implemented.** No controller, agent, installer or hub file was opened for
|
||||
editing.
|
||||
|
||||
## 3. Files (hub half)
|
||||
## 3. R-163 re-framed, and R-156's papra referral resolved
|
||||
|
||||
- `hub/internal/web/hosts.go` — `parseHostAddresses`, `hostAddressView`, `hostNetworkView`,
|
||||
`hostNetwork`, `minAgentForAddresses`, `hostDetailData` +1 key
|
||||
- `hub/internal/web/templates/host_detail_body.html` — the Network card
|
||||
- `hub/internal/web/hosts_network_test.go` (**new**)
|
||||
- `hub/internal/api/testdata/host-report.golden.json` — the cross-repo contract, moved in lockstep
|
||||
- `hub/CHANGELOG.md`, `REUSE.md`, `manifests/hub.yaml`
|
||||
- `documentation/architecture/00-capability-map.md` (new row),
|
||||
`documentation/audits/host-addresses-visible-2026-07-31.md` (new)
|
||||
**R-163 is re-framed, not closed** — as instructed. State went `WAITING-ON-OPERATOR — the ratio is a
|
||||
tier-sizing ruling` → `RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio`; "Blocked on" went
|
||||
from `the operator's sizing decision` to a pointer at R-165; owner `operator` → `CC`. The cell now says
|
||||
the sizing **question is withdrawn rather than answered**, that the row survives as the record of the
|
||||
constraint until the merge lands, and that the original finding follows unchanged. The intake ranking
|
||||
(item 4) was updated with it, and records that **R-165 inherits R-163's rank and is the highest-ranked
|
||||
item that must land before any external install**.
|
||||
|
||||
## 4. Commits pushed to `main`
|
||||
**R-156's papra referral is resolved.** The referral existed because moving a mount relocates live data
|
||||
out from under a running app; with papra deployed nowhere there is nothing to strand, so the cheaper
|
||||
leg — the template mounts `/app/app-data` — is takeable without waiting on upstream.
|
||||
|
||||
| Repo | Commit | What |
|
||||
|---|---|---|
|
||||
| felhom-agent | `14642e3` | v0.119.0 — the host report carries the box's addresses |
|
||||
| felhom-agent | `4663df7` | REPORT |
|
||||
| felhom.eu | `e07d90f` | hub v0.85.0 — Network card |
|
||||
| felhom.eu | `37f7ff6` | manifests: hub 0.84.0 → 0.85.0 |
|
||||
**The provenance is recorded with the claim, because it decides the row.** The evidence is
|
||||
`docker ps -a` on **demo-hp's guest 9201** returning empty, **supplied with the task**; this session
|
||||
**did not re-measure** — it is documentation-only and every box was fenced. The recorded scope is
|
||||
honest about its edge: it covers the one guest papra was convicted on in Campaign 10, and **no other
|
||||
customer's guest was enumerated**, so the row instructs the task that edits the template to re-check
|
||||
first. Next action on the row is the catalog edit plus `catalog_gates.py`, explicitly not done here.
|
||||
|
||||
## 5. Tests + red-proofs (hub half)
|
||||
## 4. `target-selection.md` per D-d
|
||||
|
||||
Suite **559 → 566** (+7). `go build ./... && go vet ./... && go test ./...` **rc=0**, run separately
|
||||
from the commit per standing rule 1.
|
||||
The rule at the top is now D-d: **two protected machines, everything else disposable**, with the
|
||||
correction stated as a correction — the earlier caution was costing sessions and pushing drills onto
|
||||
DooPlex. The tier table's Tier 2 row is DooPlex + Peti's cluster "and, by D-d, nothing else".
|
||||
|
||||
| Test | Asserts | Result |
|
||||
|---|---|---|
|
||||
| `TestNetwork_A_LANAddressRendered` | the LAN address + its interface reach the page | PASS |
|
||||
| `TestNetwork_B_WireGuardAllocatedAndConfirmed` | WG shown once, marked confirmed, not duplicated as a plain row | PASS |
|
||||
| `TestNetwork_C_AllocatedButBoxDoesNotHoldIt` | the drift case renders `not confirmed by the box` | PASS |
|
||||
| `TestNetwork_D_OldAgentSaysUnknownNotNone` | an old agent renders UNKNOWN, never "no addresses" | PASS |
|
||||
| `TestNetwork_E_NoPeerAllocated` | no peer → says so; the box's wg address is not silently dropped | PASS |
|
||||
| `TestNetwork_F_NoReportYet` | never-reported host says "waiting", not "no addresses" | PASS |
|
||||
| `TestParseHostAddresses_RealWire` | parses the real wire; empty/malformed degrade without panic | PASS |
|
||||
**Two consequences the decision did not name, both handled visibly rather than silently:**
|
||||
|
||||
**The report fixture is the REAL wire** — the `addresses` block copied out of
|
||||
`felhom-agent --selftest=hub` on demo-felhom running 0.119.0. A hand-written shape would have proved
|
||||
only that the parser matches my own idea of the format.
|
||||
- **`ep0` + the Hetzner Storage Boxes.** D-d's protected list has two machines and ep0 is not one, so
|
||||
the page no longer calls it Tier 2. It is **not** thereby scratch: it holds the PBS-DR datastore and
|
||||
the restic copy of a real customer's data — the only off-premises copy that exists. Read the narrow
|
||||
way (not protected, but not wipeable), using the page's own *fences-name-acts* rule, and **flagged
|
||||
in the page for the operator to confirm explicitly.**
|
||||
- **The shared "do not re-point either backup target" fence** on the two demo boxes was **downgraded
|
||||
from a prohibition to a stated cost**, because D-d makes both boxes freely reinstallable, which
|
||||
spends that reference configuration just as thoroughly — keeping the fence would have left the page
|
||||
self-contradicting. The reason survives: know you are spending the regression reference, and put the
|
||||
box back.
|
||||
|
||||
| # | Mutation | Observed | Reverted |
|
||||
|---|---|---|---|
|
||||
| A | `hostDetailData` ships an empty `hostNetworkView{}` (the inert-seam shape) | `FAIL … the LAN address is not on the host page — the feature shows nothing` | yes |
|
||||
| B | `WGConfirmed` set unconditionally from the allocation | `FAIL … an allocated-but-unheld peer renders as healthy — the drift is invisible` | yes |
|
||||
| D | delete the `AgentTooOld` branch | `FAIL … an old agent's silence is not explained` | yes |
|
||||
Also corrected while in the file: the *fences-name-acts* example cited the fence this edit removed, and
|
||||
demo-hp's access line asserted "no baked SSH key" — which **R-129** records as measured false on
|
||||
2026-07-31. It now points at R-129 instead of sending the next session to the hub vault for a
|
||||
credential it may not need.
|
||||
|
||||
**Red-proof A took two attempts, and the first one proved nothing.** The initial mutation
|
||||
(`_ = network` inside a map literal) did not compile, so the "failure" was a build error, not a test
|
||||
result. Recorded because a red-proof that fails to build is indistinguishable from one that lands if
|
||||
you only read the exit code.
|
||||
## 5. The maintenance rule
|
||||
|
||||
## 6. Deployed
|
||||
In two places, as specified: the `STATUS.md` header block, and a new **`## End-of-session checklist`**
|
||||
in `CLAUDE.md` — which also gathers the couplings that were previously scattered (CHANGELOG + REPORT,
|
||||
REUSE, the capability map's own end-of-session line, S-1's architecture coupling) and closes with *a
|
||||
finding goes in `OPEN-ITEMS.md` first, never only in a report, an audit or `STATUS.md`*.
|
||||
|
||||
Image `gitea.dooplex.hu/admin/felhom-hub:0.85.0` built on DooPlex and pushed; `manifests/hub.yaml`
|
||||
bumped in git; ArgoCD hard-refreshed and **deliberately synced** (never `kubectl set image`).
|
||||
`CONTEXT.md` gained a header block stating why it and `STATUS.md` are separate — same subjects,
|
||||
different readers, and `STATUS.md` holds nothing of its own. `STATUS.md` says the same from its side.
|
||||
|
||||
- ArgoCD app `felhom`: **Synced / Healthy**
|
||||
- `deploy/hub` image: **`gitea.dooplex.hu/admin/felhom-hub:0.85.0`**
|
||||
- Rollout: `deployment "hub" successfully rolled out` (rc=0)
|
||||
- Startup log: `2026/07/31 08:50:57 [INFO] felhom-hub 0.85.0 starting`
|
||||
## 6. What could not be translated into plain language
|
||||
|
||||
**One sequencing note worth keeping:** the first sync fired before ArgoCD had re-read git, so it
|
||||
reported `Synced/Healthy` while still running 0.84.0 — a green pair that meant nothing. The fix is to
|
||||
wait until `.status.sync.revision` equals the pushed commit **before** trusting the sync, which the
|
||||
final run did.
|
||||
Asked for explicitly, because an untranslatable row usually means the row itself is unclear.
|
||||
|
||||
## 7. Live validation
|
||||
- **R-29** — *"gates are enforced nowhere"*. The class is stateable ("we have checks nobody runs"), but
|
||||
its instances are four differently-broken scripts across two repos with no shared consequence, so
|
||||
every plain sentence either says nothing or misstates one instance. **R-161 is its readable
|
||||
fragment**, which is why R-161 is on the page and R-29 is not.
|
||||
- **R-123 / R-125** — process findings about how the register and how tests are written. Real, and they
|
||||
belong on the register; there is no customer-visible symptom to lead with, so they have no honest
|
||||
first sentence for this page. They are not "broken" in the operator's sense.
|
||||
- **R-133 (the plaintext break-glass credential)** — translatable, and left off only for space. It is
|
||||
the strongest candidate for the next update if something else closes.
|
||||
- **R-115 vs R-110** — separate rows, one plain-language paragraph. Merged into a single "Waiting on
|
||||
you" bullet carrying both pointers, because two adjacent bullets about publishing read as one item
|
||||
the operator has already half-decided.
|
||||
|
||||
**Method: endpoint-level** (no browser on DooPlex), against the deployed hub, cross-checked
|
||||
against `ip addr` read directly over SSH on both boxes.
|
||||
**A register defect found while reading, filed here because the fix is not mine to guess: `R-133` is
|
||||
used TWICE** — `OPEN-ITEMS.md:80` (duplicate `domain` values accepted by the hub) and `:86` (the
|
||||
plaintext break-glass credential). Two different findings, one ID, both `READY`. One needs renumbering,
|
||||
and which one is the operator's call since both are cited from elsewhere (`CONTEXT.md` S-4 cites the
|
||||
credential one).
|
||||
|
||||
| Host | Agent | Network card, live | Matches ground truth? |
|
||||
|---|---|---|---|
|
||||
| `demo-felhom-8363b5` | 0.119.0 | WG **10.77.0.2 confirmed**; `vmbr0 192.168.0.162/24`, `tailscale0 100.70.170.35/32` + ULA | **yes, exactly** |
|
||||
| `demo-hp-bb76ea` | 0.119.0 | WG **10.77.0.3 confirmed**; `vmbr0 192.168.0.87/24`, `tailscale0 100.76.96.79/32` + ULA | **yes, exactly** |
|
||||
| `drill-r50-0a4f9a` | 0.113.0 | WG 10.77.0.4 **allocated**; *"agent (0.113.0) does not report its addresses — they are unknown, not absent. Needs 0.119.0 or newer."* | correct UNKNOWN rendering |
|
||||
| `sess-f-2670b5` | old | same UNKNOWN rendering | correct |
|
||||
## 7. Files changed
|
||||
|
||||
Correctly absent from every card: `lo`, `vmbr9`'s `169.254.253.1` island literal, every `fe80::`.
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `STATUS.md` | **new** — the operator page |
|
||||
| `CONTEXT.md` | S-5 (D-a … D-d); header note on the `STATUS.md` separation |
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | R-165/166/167 filed; R-163 re-framed (state, blocked-on, owner, ranking); R-156's referral resolved with its provenance |
|
||||
| `documentation/runbooks/target-selection.md` | D-d rule; tier table; ep0; the demo-box fence; two stale lines |
|
||||
| `CLAUDE.md` | new `## End-of-session checklist`, carrying the `STATUS.md` maintenance rule |
|
||||
|
||||
**A fifth host 404'd and it was not a bug.** `sess-g-e687c3` appeared in a host list I read 30
|
||||
minutes earlier and returned 404 when I fetched its page; the concurrent session had deleted it as
|
||||
part of its scratch-customer cleanup. Chased to ground before reporting rather than filed as a defect.
|
||||
|
||||
## 8. Not yet validated / not covered
|
||||
|
||||
- **A box with two LAN bridges** — handled by design (one row per interface) and unit-tested; no such
|
||||
box exists to observe.
|
||||
- **A real WireGuard drift** — the `not confirmed` branch is unit-tested and red-proofed but never
|
||||
seen in production, because no live peer is currently unapplied.
|
||||
- **IPv4-only / IPv6-only hosts** — every box here is dual-stacked via tailscale's ULA.
|
||||
|
||||
## 9. Teardown
|
||||
|
||||
This run provisioned nothing — no guest, no VM, no scratch customer, no external resource. The only
|
||||
mutations are two in-place agent upgrades (previous binaries retained as `.bak-0.118.1` on both
|
||||
boxes), the Day-0 manifest agent vouch, and the hub image bump.
|
||||
|
||||
## 10. Backlog
|
||||
|
||||
- **Opened: none.** **Closed: none.**
|
||||
- **R-115 / R-121 touched, not closed.** This release walked all three legs deliberately —
|
||||
deployed, **published** (round-trip verified) and **vouched** (read back) — so no new drift
|
||||
instance was created. The finding those rows record is that nothing *notices* when the legs
|
||||
diverge, which is unchanged. One small contribution: the Network card now surfaces
|
||||
"this host's agent is older than 0.119.0" per host, which is a per-field hint, not the
|
||||
installed-vs-vouched checker R-121 asks for.
|
||||
- **Concurrent session:** every path was staged explicitly; no foreign WIP was swept into any commit.
|
||||
Nothing was built, deployed, published or touched on any host. Every claim about the register above is
|
||||
a claim about pushed source in this repo at the commit below.
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Updated 2026-08-02.**
|
||||
|
||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
||||
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
||||
> which is technical state written for Claude Code — keep the two separate. **Maintenance:** update
|
||||
> at the end of every session in which something shipped, broke, or was decided. One screen; cut
|
||||
> items rather than extend it.
|
||||
|
||||
## What works right now
|
||||
|
||||
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
||||
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
||||
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
||||
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
||||
restore files and app data from the drive alone. Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
|
||||
**After a power cut, an app can stay switched off — and nothing says so.** The machine looks for apps
|
||||
that didn't come back, but looks too early and never again. In one case it decides the customer
|
||||
switched it off deliberately, so it isn't even counted as down. *(R-157)*
|
||||
|
||||
**The off-site copy can be erased by the machine that made it** — the credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
*(R-95, R-87)*
|
||||
|
||||
**Three apps out of fifty-three kept their data where backups never looked.** They reported healthy;
|
||||
the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is
|
||||
installed nowhere. *(R-156)*
|
||||
|
||||
**Local backups get 20 GB while apps get 50 GB.** An app that outgrows the smaller space stops being
|
||||
backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is
|
||||
lost: the last good copy is kept intact. *(R-163)*
|
||||
|
||||
**When that happens, only one page says so** — no email, no alert. The page that answers "is this app
|
||||
backed up?" is the one that stays silent. *(R-158)*
|
||||
|
||||
**The check that catches this needs a person to remember it.** One command, run by hand; nothing
|
||||
refuses a change that skipped it. *(R-161)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** the last app whose data was never saved; today's decisions written down.
|
||||
- **Next:** merging the small backup partition into the large one, with the drive-filling warning and
|
||||
the backup-failure alert in the same step.
|
||||
- **After:** rebuilding how the machine records whether an app is meant to be running.
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **How a new version reaches a machine.** Pushing the installer publishes it — half a minute later
|
||||
every new machine downloads it, with no staging and no way back but another push. And publishing is
|
||||
a step we remember rather than one the release performs, forgotten twice: a fix can be live here
|
||||
and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to
|
||||
settle both. *(R-110, R-115)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-02** — Decided: the 20 GB backup partition goes away and shares space with app data. That
|
||||
changes the disk layout, so it happens before any machine is installed outside the house.
|
||||
- **2026-08-02** — Decided: only this machine and the tester's box are protected; every other box,
|
||||
both demo boxes included, may be broken or reinstalled freely.
|
||||
- **2026-08-02** — Two of the three apps that never saved their data are fixed; this page created.
|
||||
@@ -84,15 +84,18 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-137** | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with "<domain>"` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC |
|
||||
| **R-138** | **A shared-zone `cf_api_token` is a zone-wide DNS-write capability on a customer's box** — written 0600 to `/opt/docker/stacks/traefik/.env` (`controller/internal/infra/infra.go:123`) | READY (S) | — | Today each box holds a token for a zone nobody else uses, so the blast radius is one customer. Under a shared customer zone, one compromised tester box could repoint every other tester's DNS. The ACME path is already switchable — an empty token selects HTTP-01 (`traefik.yml.tmpl`) — so the fix is policy plus a guard that refuses to hand a shared-zone customer a zone-scoped token. Same audit §5.2 | CC |
|
||||
| **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC |
|
||||
| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. Next action is papra's referral only. See R-161 (nothing runs the gate) and R-159/R-160 | CC |
|
||||
| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. **THE REFERRAL IS RESOLVED, 2026-08-02 — papra is deployed NOWHERE, so the template fix strands nothing and can be applied.** The referral existed because changing where the volume mounts moves live data: an installed papra writes `db.sqlite` into the container's writable layer, and a remount relocates the path out from under it. With no instance deployed there is no live data to move, so the cheaper leg — **the template mounts `/app/app-data`** — is takeable directly, without waiting on upstream to adopt `/app/data`. **Provenance, stated because it decides the row:** the observation is `docker ps -a` on demo-hp's **guest 9201** returning empty, supplied with the 2026-08-02 task; **this session did not re-measure** (documentation-only, every box fenced). **Scope of that evidence, honestly:** it covers guest 9201 — the guest papra was convicted on in Campaign 10 — and **no other customer's guest was enumerated**, so a re-check belongs in the task that edits the template, before it edits it. **Next action: apply the template fix in `app-catalog-felhom.eu` and re-run `scripts/catalog_gates.py`** (deliberately not done here — that repo was out of scope for this task). See R-161 (nothing runs the gate automatically) and R-159/R-160 | CC |
|
||||
| **R-157** | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms, and a settle-condition fix closes only one.** **(A)** the sweep runs ~5 s after controller start (`cmd/controller/main.go:236`) while docker is still restoring, sees "no boot-orphaned apps", and never re-checks — **intermittent, 3 of 6 hard resets**; the predicate itself is sound (`bootrecon.go:100`, and `IsDownState` includes `StateDegraded`, `manager.go:55`). **(B)** an interruption that lands mid-compose leaves the stack with **zero containers**, which is exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp check reported **`0 currently down`** while a `deployed: true` app was not running: **silent on every channel**. | **READY (M)** | — | **Two fixes, not one.** A: a settle condition before the single observation, or one re-evaluation at the end of `deadAppBootGrace`, still bounded to the same attempts. B: needs a signature that distinguishes "customer stopped this" from "interrupted between containers" — B is F-CRIT-1's shape (`CLAUDE.md` false-invariant #4) reached via hard reset rather than quiesce. Test the **consequence** (app running again before the grace expires), not the mechanism. Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` | CC |
|
||||
| **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC |
|
||||
| **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `<project>_<name>` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC |
|
||||
| **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |
|
||||
| **R-161** | **The volume-persistence gate is enforced by CONVENTION, not automatically.** The catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none exists). | **REDUCED SCOPE — open** (operator ruling 2026-08-02) | a second person touching templates | **RULED. Both obvious enforcement points were rejected for measured reasons.** *Controller-side at template load:* rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — **it would pass on the exact defect it exists to catch**; the property is decidable only at runtime. *CI:* rejected for now — neither repo has any, and there are no users yet. **SHIPPED instead** (`app-catalog-felhom.eu` `fd7747d`): `scripts/catalog_gates.py`, ONE entry point running all three gates, non-zero exit on any failure, **mandated in the catalog's `CLAUDE.md`** the way `site_gates.py` is. Rationale for the record: of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md — `site_gates.py` is run, R-29's three orphans are named nowhere and have stopped nothing. **What remains open is only the automatic half:** this is convention, run by a person, and that is sufficient while one person touches templates. Revisit when a second does | operator |
|
||||
| **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC |
|
||||
| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | the operator's sizing decision | **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 | operator |
|
||||
| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio** | — (the sizing question is answered; the work is **R-165**) | **RE-FRAMED, NOT CLOSED (operator decision D-a, 2026-08-02 — `CONTEXT.md` S-5).** The row asked *what ratio should `mp1` be?* and that question is **withdrawn rather than answered**: `mp1` is merged into `mp0` so local recovery units share the app-data area and the ceiling stops existing — a bigger number is the same wall further away. **This row stays open as the record of the constraint** (what `mp1` is for, what it gates, and the measured 2× DB-app unit size) **until the merge lands**, because until then every consequence below is still live on every box. **The work is R-165; the warning that must ship with it is R-167.** Original finding, unchanged, follows. **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 (widened to R-167) | CC |
|
||||
| **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
|
||||
| **R-165** | **Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.** Operator decision **D-a**, 2026-08-02 (`CONTEXT.md` S-5). Local recovery units share the app-data area instead of holding their own fixed ceiling, so the wall R-163 describes is removed rather than moved further away. Guest 9201 on demo-hp ships `mp0 50G / mp1 20G` today | **READY (M) — NEW 2026-08-02** | — | **Two conditions travel WITH the decision and are not optional.** **(1) Before any external install.** It changes the **disk layout**, so it is a fresh-install shape while there are no external boxes and a per-box migration after — and the decision's cheapness is entirely a function of that ordering. **(2) It removes a wall that currently fails safely**, so **R-167** (D-c: fill warning + failure alert) lands in the same step, never after: today an app that outgrows `mp1` is refused per app with the last good unit preserved byte-identical (R-158's measurement), and after the merge the same overflow consumes the space the app itself is using. Touches the installer/agent guest shape (`--sysdata-grow` defaults to **0** and is not derived from the physical drive, `felhom-agent/cmd/felhom-agent/main.go:178`) and the golden. **Does NOT close R-163** — that row is the record of the constraint and stays open until this lands | CC |
|
||||
| **R-166** | **App state gets a desired/observed model with its own store.** Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5). Three kinds of information, deliberately separated: **desired** (running, or stopped because the customer said so) stays in `app.yaml`; **in-flight operations** (a backup started and never reported finishing) are **persisted** — written before the operation and after it — so an interruption is visible after a reboot; **observed** (unhealthy, restarting, drive gone) is **not** persisted and is rebuilt by looking. The last two live in a store **separate from `app.yaml`**, so frequent writes never touch a file holding deploy settings and secrets | **BLOCKED — NEW 2026-08-02** | two facts to establish before it is specced | **NOT a spec yet, deliberately.** **Establish first:** (a) whether the crash-safe journal already in the backup code covers the in-flight case — *it may already exist and merely be unwired, which has been the answer four times in this project* (`CLAUDE.md` "seam built but never wired"); (b) whether the existing SQLite store is reachable from this code path. **Binding safety rule, verbatim from the decision:** *losing the state store must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the worst acceptable outcome is re-running a backup that already ran.* The controller must also check **every container of an app**, so a partly-dead app cannot read as healthy. **This is the durable fix for R-157 mechanism B** (zero containers reading as a deliberate stop) and for F-CRIT-1's class; R-157 keeps its own row because it wants a bounded fix now, not this rework | CC |
|
||||
| **R-167** | **Storage monitoring and backup alerts.** Operator decision **D-c**, 2026-08-02 (`CONTEXT.md` S-5). Two pieces: a warning to the **customer** as a drive fills, **before** anything fails; and a failure alert to the **hub/operator** when any backup fails for **any** reason, lack of space included | **READY (M) — NEW 2026-08-02** | — | **Prerequisite for R-165** — D-a removes a wall that currently fails safely, so this ships in the same step, not after. **Subsumes and widens R-158**, which is the same alert one tier down (`NotifyBackupFailed` exists and the hub allowlists `backup_failed`; only the off-box/NAS leg calls it, `cmd/controller/main.go:659`). R-158 stays as the named seam and its proposed `unitNotify` shape is the concrete first leg — **do not file a second row for the same wire.** The customer half is new: nothing today warns before a fill, and R-158's measurement is that when it happens exactly one page says so | CC |
|
||||
|
||||
## Why the TOP READY rows rank this way
|
||||
|
||||
@@ -132,7 +135,10 @@ unrelated finding, which is why the sweep's proposals were renumbered to R-159
|
||||
ran the gate; it now has a mandated entry point, so R-156's residue is the larger remaining item.)*
|
||||
4. **R-163** — a real ceiling that silently caps local backup once an app outgrows `mp1`, and it gates
|
||||
Tier-2 and Tier-3 as well. Ranked below the above only because **overflow itself is safe today** —
|
||||
it refuses per app and preserves the last good unit byte-identical. Waiting on the operator's ratio.
|
||||
it refuses per app and preserves the last good unit byte-identical. **RE-FRAMED 2026-08-02:** no
|
||||
longer waiting on a ratio — decision **D-a** merges `mp1` away, so the row is now the record of the
|
||||
constraint and the work moves to **R-165** (with **R-167** shipping in the same step). R-165 inherits
|
||||
this rank; it is the highest-ranked item that must land **before any external install**.
|
||||
5. **R-158** — the gap that makes R-163 dangerous: cross the size line and **one page** tells you. On
|
||||
its own it is a notification gap, not a silent failure, which is why it sits here and not higher.
|
||||
6. **R-164** — blocked on a predicate, no customer impact today; it only becomes urgent if the unit
|
||||
|
||||
@@ -2,28 +2,48 @@
|
||||
|
||||
> Read before picking a machine for a drill, a destructive test, or a throwaway VM. It answers one
|
||||
> question: *what is safe to lose.* Reachability is in `CLAUDE.md`; hardware in `operations/nodes.md`.
|
||||
> Added 2026-07-30.
|
||||
> Added 2026-07-30. **Revised 2026-08-02 by operator decision D-d** (`CONTEXT.md` S-5).
|
||||
|
||||
## The rule
|
||||
|
||||
> **Start at Tier 0 and work down only if Tier 0 genuinely cannot host the work.**
|
||||
> **A Tier 2 machine is used only when a task says so explicitly** — never by inference from what was
|
||||
> **Two machines are protected: `DooPlex` and Peti's box. Everything else is disposable.**
|
||||
> Operator decision **D-d**, 2026-08-02. DooPlex because it holds Gitea, the hub, the backups and the
|
||||
> registry — everything else rebuilds from it, and it rebuilds from nothing. Peti's box because there
|
||||
> is a real person behind it. **Every other box, both demo boxes included, may be broken or
|
||||
> reinstalled freely.**
|
||||
>
|
||||
> **This is a correction, not a relaxation.** The earlier posture was costing whole sessions to
|
||||
> caution and pushing drills onto DooPlex — the one machine that should never host them. If you are
|
||||
> weighing whether a demo box can take a destructive test: it can. **Reach for a Tier 0 box first and
|
||||
> do not ask.**
|
||||
>
|
||||
> Still true: **start at Tier 0 and work down only if Tier 0 genuinely cannot host the work**, and a
|
||||
> protected machine is used only when a task says so explicitly — never by inference from what was
|
||||
> not forbidden. An absent fence is not permission. If no tier fits, **stop and ask.**
|
||||
|
||||
Fences name **acts**, not machines. "Do not re-target demo-hp's backup target" and "do not use demo-hp
|
||||
to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine
|
||||
Fences name **acts**, not machines. "Do not destroy demo-hp's `drill-r50` fixture" and "do not use
|
||||
demo-hp to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine
|
||||
prohibition as covering the act it names and nothing more.
|
||||
|
||||
| Tier | Meaning | Machines |
|
||||
|---|---|---|
|
||||
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
|
||||
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
|
||||
| **2 — precious. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **ep0** + the Hetzner Storage Boxes |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
|
||||
|
||||
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
|
||||
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
|
||||
thing under test, the source of truth for it, and the backups, at once.
|
||||
|
||||
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
|
||||
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
|
||||
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
|
||||
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
|
||||
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
|
||||
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
|
||||
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
|
||||
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
|
||||
|
||||
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
|
||||
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
|
||||
|
||||
@@ -38,20 +58,24 @@ felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat
|
||||
**`/mnt/nvme-1tb`, at its root** (a subdirectory fails the agent's `exactMount` check → storage reads
|
||||
`disconnected` forever). **That warning is about one storage, not the box.** `/mnt/nvme-1tb` is also
|
||||
the `felhom-backup` target and the enrolled user-data drive, so remove scratch storages when done.
|
||||
- **Forbidden:** do not re-target its backup configuration (below). Do not destroy or unblock
|
||||
**`drill-r50` (VM 300)** — the only drift fixture (R-93). No baked SSH key; access is the hub-vaulted
|
||||
G1 break-glass password (R-61).
|
||||
- **Forbidden:** do not destroy or unblock **`drill-r50` (VM 300)** — the only drift fixture (R-93).
|
||||
(Access: the docs say no baked SSH key and G1 break-glass, but a key authenticated on 2026-07-31 —
|
||||
**R-129**, unresolved.)
|
||||
|
||||
### `demo-felhom` — N100 · **Tier 0**
|
||||
|
||||
- **Freely:** create/destroy guests and scratch customers; reinstall the box.
|
||||
- **Care:** it carries the **PBS-DR / offsite tier** (demo-hp has none), so it is the Tier 0 box whose
|
||||
backup chain a test can actually disturb. Prefer demo-hp, per the 2026-07-25 ruling.
|
||||
- **Forbidden:** do not re-target its backup configuration.
|
||||
|
||||
**Both Tier 0 boxes, one shared prohibition and its reason:** do not re-point either backup target.
|
||||
They are the only two **correctly configured** boxes, so they are the regression path new installer
|
||||
logic is measured against — losing that reference costs more than the box does.
|
||||
**Both Tier 0 boxes — the shared backup-target fence is DOWNGRADED to a cost, 2026-08-02 (D-d).** It
|
||||
read *"do not re-point either backup target"*, because these are the only two **correctly configured**
|
||||
boxes and therefore the regression path new installer logic is measured against. D-d makes both boxes
|
||||
freely breakable and reinstallable, which loses that reference just as thoroughly — so the fence was
|
||||
inconsistent with the decision and is not kept as a prohibition. **What survives is the reason:**
|
||||
re-pointing (or reinstalling) costs the reference configuration, so know that you are spending it and
|
||||
put the box back. If both are spent at once there is no correctly-configured box left to compare
|
||||
against.
|
||||
|
||||
### `DooPlex` — 192.168.0.180 · **Tier 2**
|
||||
|
||||
@@ -71,7 +95,7 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
|
||||
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
|
||||
enrolled host. No access route from DooPlex, and nothing here needs one.
|
||||
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2**
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
|
||||
|
||||
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
||||
path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
|
||||
|
||||
Reference in New Issue
Block a user