From e994bf35d2908f8d65486a252a1a5d8963cc397b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 2 Aug 2026 14:20:29 +0200 Subject: [PATCH] STATUS.md: a plain-language operator page, and today's four decisions recorded MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documentation only — no code, no box, no build. STATUS.md (repo root, 652 words / 67 lines): what works · what's broken · what we're working on · waiting on you · changed since. A VIEW of OPEN-ITEMS.md, holding nothing of its own; not CONTEXT.md, and both files now say why they stay separate. No R-n is the subject of a sentence — identifiers are bracketed pointers only. CONTEXT.md S-5 records the four operator decisions taken 2026-08-02 (D-a … D-d), none of them implemented: D-a merge mp1 into mp0 rather than resize it — before any external install, and D-c ships in the same step → R-165 D-b desired/observed app state in its own store, with the state-store safety rule verbatim → R-166 (BLOCKED) D-c customer fill warning + operator backup-failure alert → R-167 D-d only DooPlex and Peti's box are protected → target-selection.md R-163 RE-FRAMED, not closed: the sizing question is withdrawn rather than answered; the row survives as the record of the constraint until R-165 lands. R-156's papra referral RESOLVED — deployed nowhere, so the template fix strands nothing; the docker ps evidence is recorded with its provenance and its scope limit. target-selection.md: two protected machines, everything else disposable. ep0 is no longer Tier 2 but is not scratch (it holds the only off-premises copy of real customer data) — flagged for explicit operator confirmation. The demo-box backup-target fence drops from prohibition to stated cost, because D-d spends that reference anyway. CLAUDE.md gains an End-of-session checklist carrying the STATUS.md maintenance rule and "a finding goes in OPEN-ITEMS.md first". --- CLAUDE.md | 14 ++ CONTEXT.md | 52 ++++ REPORT.md | 264 +++++++++++---------- STATUS.md | 66 ++++++ documentation/backlog/OPEN-ITEMS.md | 12 +- documentation/runbooks/target-selection.md | 52 ++-- 6 files changed, 314 insertions(+), 146 deletions(-) create mode 100644 STATUS.md diff --git a/CLAUDE.md b/CLAUDE.md index ffa2fa4..9d8ffb7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -102,6 +102,20 @@ pushes; **you (Claude Code) implement**. A file being open in the editor is NOT explicit paths only, `git pull --rebase` before every push, and do not run two writing sessions on one clone (use `git worktree` if truly needed). +## End-of-session checklist + +- **`CHANGELOG.md` + `REPORT.md`** per the rule above, in every repo touched. +- **`REUSE.md`**, if a shared helper or pattern moved (same commit). +- **The capability map** (`documentation/architecture/00-capability-map.md`), if a capability's status + changed — with its new evidence citation. +- **The architecture doc** that owns any changed contract (S-1, `CONTEXT.md`). +- **Root `STATUS.md`** — *update it at the end of every session in which something shipped, broke, or + was decided. It is a view of `documentation/backlog/OPEN-ITEMS.md` — **nothing may exist only + there**. One screen; cut items rather than extending it.* It is written for the **operator** in + plain language, and is deliberately **not** `CONTEXT.md` — do not consolidate the two. +- **A finding goes in `OPEN-ITEMS.md` first**, never only in a report, an audit or `STATUS.md`. Four + items in this project were minted in a spike doc and lost (R-153/154/155, R-156/157). + ## Tech stack (Hub) - **Language:** Go (stdlib `net/http` + `html/template`, no frameworks). **DB:** SQLite via diff --git a/CONTEXT.md b/CONTEXT.md index 96375c4..b8773bf 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,6 +7,14 @@ > Read it first; `ROADMAP.md` holds the full history and reasoning behind each item. **`REPORT.md` is > per-session and overwritten — nothing durable may live only there.** +> **This file is NOT `STATUS.md`, and the two must not be consolidated** (2026-08-02). `CONTEXT.md` +> is technical working state written for Claude Code: rulings with their mechanisms, file paths, +> version numbers, the reasoning a session needs before it touches code. Root `STATUS.md` is written +> for the **operator** — technical, but not in the code and not a reader of the audits — in plain +> language, one screen, no identifiers in the prose. Same subjects, different readers; merging them +> would make one of the two audiences stop reading. `STATUS.md` is also a **view of `OPEN-ITEMS.md`** +> and holds nothing of its own; this file does hold its own content, namely the standing rulings below. + ## Standing rulings **S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81).** @@ -79,6 +87,50 @@ Five decisions were deliberately **left open for the operator** and are recorded stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with the guest it backs up. Gaps minted the same session: **R-102 … R-108**. +**S-5 — four operator decisions taken in discussion on 2026-08-02, recorded before anything is +built.** They existed only in conversation, which is the condition the standing rules were written +against. Labels are the ones used in the discussion (**D-a … D-d**) and are deliberately kept +distinct from S-3's `D1…D6`. **None of a, b or c is implemented by the session that recorded them**; +their work is carried as backlog rows (**R-165, R-166, R-167**), not as prose here. + +- **D-a — the dedicated backup partition is MERGED AWAY, not resized.** The appliance's system drive + carries the OS, a **50 G** docker area (`mp0`) and a separate **20 G** area holding local recovery + units (`mp1`; demo-hp's guest 9201 ships exactly that). The decision is to **merge the backup area + into the large data area** so local backups share space with app data and the ceiling stops + existing — deliberately *not* to pick a larger number, because a larger number is the same wall + further away. **Two conditions, recorded with the decision, not as commentary:** (1) it changes the + **disk layout**, so it must land **before any external install** — after that, every deployed box + is a migration; (2) it **removes a wall that currently fails safely**, so the storage monitoring of + **D-c** must ship **with** it, never after. Closes the sizing question in **R-163**, which is + **re-framed rather than answered with a ratio** and is not closed until the merge lands. Work: + **R-165**. +- **D-b — app state gets a desired/observed model with its own store.** Three kinds of information, + deliberately separated: **desired** (running, or stopped because the customer said so) stays in + `app.yaml` with that app's other settings; **in-flight operations** ("a backup started and has not + reported finishing") are **persisted**, written before the operation starts and after it ends, so + an interruption is visible after a reboot; **observed** (unhealthy, restarting, drive gone) is + **not** persisted and is rebuilt by looking. The last two live in a **state store separate from + `app.yaml`**, so frequent writes never touch a file holding deploy settings and secrets. + **Binding safety rule, verbatim:** *losing the state store must never cause an app to be deleted, + restarted wrongly, or reported healthy when it is not — the worst acceptable outcome is re-running + a backup that already ran.* The controller must also check **every container of an app**, so a + partly-dead app cannot read as healthy (F-CRIT-1's shape). **Two things to establish before this is + specced, not now:** whether the crash-safe journal already in the backup code covers the in-flight + case — *it may already exist and merely be unwired, which has been the answer four times* — and + whether the existing SQLite store is reachable from this code path. Work: **R-166**. +- **D-c — storage monitoring and backup alerts.** Two pieces: a warning to the **customer** as a + drive fills, **before** anything fails; and a failure alert to the **hub/operator** when any backup + fails for any reason, lack of space included. **Prerequisite for D-a** — see that decision's + condition (2). Subsumes and widens **R-158**, which is the same alert one tier down; R-158 stays as + the named seam and is not duplicated. Work: **R-167**. +- **D-d — which machines are protected: DooPlex and Peti's box, and nothing else.** DooPlex because + it holds Gitea, the hub, the backups and the registry — everything else rebuilds from it. Peti's + box because there is a real person behind it. **Every other box, both demo boxes included, is + disposable and may be broken or reinstalled freely.** The earlier caution was costing sessions and + pushing work onto DooPlex — the one machine that should never host it — so the ruling is a + correction, not a relaxation. `documentation/runbooks/target-selection.md` updated the same session + (2026-08-02); no backlog row, the decision *is* the change. + - **2026-07-27 — R-85: the DR tier is restore-tested UNATTENDED, and its failure is HEARD.** agent v0.104.0 + hub v0.77.0. R-82 proved a PBS restore round-trip **by hand**; three defects kept the unattended path from covering it. (a) `buildRestoreTestScheduler` built ONE runner on diff --git a/REPORT.md b/REPORT.md index fce0624..5d8a9e5 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,162 +1,168 @@ -# REPORT — Hub v0.85.0: the Network card (2026-07-31) +# REPORT — `STATUS.md` created, and the 2026-08-02 operator decisions recorded (2026-08-02) -**Overwritten** per the standing rule. Pairs with **agent v0.119.0** (see `felhom-agent/REPORT.md`); -neither half is useful alone. Live evidence: -`documentation/audits/host-addresses-visible-2026-07-31.md`. +**Overwritten** per the standing rule. The prior contents (hub v0.85.0 Network card + v0.86.0 Copy +without reveal, 2026-07-31) have their durable record in `hub/CHANGELOG.md` and +`documentation/audits/host-addresses-visible-2026-07-31.md`; nothing was lost by this overwrite. -> The earlier v0.84.0 work this file previously described (break-glass Console access card) shipped, -> deployed and validated in the same session; its durable record is `hub/CHANGELOG.md` v0.84.0 plus -> `CONTEXT.md` ruling **S-4** and `documentation/backlog/` **R-133**. +**Class: documentation only.** No code, no template, no box, no build. Repo `felhom.eu` only — +`app-catalog-felhom.eu` was read for context and **not** modified. **No `CHANGELOG.md` entry exists +for this change and none is missing:** this repo has no root changelog, only per-area `hub/`, +`scripts/`, `website/` (`CLAUDE.md`), and this session touched none of those areas. -## 0. Follow-up shipped the same session — hub v0.86.0 (Copy without reveal) +Baselines: `felhom.eu` @ `260a8f6`, `app-catalog-felhom.eu` @ `fd7747d`. Part 1 was derived by reading +the register's rows and both ranking sections, not from memory. -**The operator hit a real defect in the v0.84.0 Console access card and it cost a login.** Copy was -`disabled` until a Reveal, so clicking it did **nothing, silently**; the clipboard kept its previous -contents — **another host's console password** — which was pasted into demo-hp's PVE login. It failed -with `password check failed for user (root)`, which reads exactly like a stale credential and sends -you diagnosing the wrong thing. +--- -Diagnosis, read-only first: the vaulted password **matched the box's `/etc/shadow` hash**, the account -was unlocked and non-expiring, no TFA, no `pam_faillock` — and then it **minted a real PVE ticket** -(`POST /api2/json/access/ticket` → HTTP 200, `root@pam`, 367-char ticket). So the credential was -always good; only the clipboard was wrong. **That also closes the leg §7 of this report listed as -operator-only** — the capability-map row now records it. +## 1. `STATUS.md` — the file -**v0.86.0** makes Copy work without revealing (the *safer* default — the secret never renders, so it -cannot be shoulder-surfed or screenshotted) and closes three silent-failure branches in one eight-line -function: the disabled no-op, a missing `navigator.clipboard`, and an **ignored `writeText()` -rejection** that let the operator believe a refused write had succeeded. The success message now names -the host, because the clipboard is fleet-wide and "copied" alone cannot say for which box. +Root of `felhom.eu`, so it is the first thing visible. Sections in the specified order: what works · +what's broken · what we're working on · waiting on you · changed since last update. -Tests 566 → 568, both pinning the regression; red-proof: re-adding `disabled` reproduces the shipped -bug. Deployed and live-verified (`Synced/Healthy`, image `:0.86.0`, button served without `disabled`, -retrieval URL defined exactly once, all four outcome messages present). Commits `670ec35` + `80f4739`. +**Word count: 652 total, 581 excluding the header block** (`wc -w`; the header carries the +view-not-source, not-`CONTEXT.md` and maintenance rules, which the spec requires). **That is over the +~500 target and it is a deliberate miss, stated rather than hidden.** Five passes took it from 819 to +652. Getting under 500 needed either dropping a mandated item or dropping the off-site-credential line +— the register's **top-ranked** open item and the largest customer-data exposure on it. Cutting the +biggest data risk to save forty words is the wrong trade on a page whose job is to show the operator +what is at stake. It is 67 lines and fits a screen. **If the operator disagrees, the line to cut is +the R-95/R-87 one** and the page drops to ~545. -## 1. Baselines +Content, in the operator's ranking: an app can stay off after a power cut, silently (R-157) · the +off-site copy can be erased by the box that wrote it, and has never been restored from (R-95, R-87) · +three of fifty-three apps saved data where backups never looked (R-156) · 20 GB of backup space +against 50 GB of apps (R-163) · when that trips, one page says so and nothing alerts (R-158) · the +checker exists but a person has to remember it (R-161). -| Repo | `main` @ start of this half | Version before → after | +**Constraints honoured:** no `R-n` is the subject of any sentence — every identifier is a bracketed +pointer at the end of a line; no file paths, function names or version numbers appear; every broken +item is stated as what a customer or the operator would notice. Shipped, watching and +blocked-on-a-predicate rows (R-159, R-160, R-162, R-164) are absent by design. + +**One deviation, flagged per standing rule 4.** "Waiting on you" is specified as *decisions only*, and +it carries one non-decision: **the hub password needs rotating** (R-132, owner Viktor). It is the only +thing on the register waiting on the operator with a live credential consequence, and omitting it from +the operator's own page to honour a section rule would be the letter over the point. It is labelled +*"a job, not a decision"* so the section's shape is not quietly eroded. + +## 2. The decisions — where each one went + +All four are in **`CONTEXT.md` as standing ruling S-5**, labelled **D-a … D-d** as in the discussion +and deliberately kept distinct from S-3's `D1…D6`. Open work is carried as backlog rows, per the +existing convention — no new home was created for either. + +| Decision | Recorded | Work | |---|---|---| -| felhom.eu (hub) | `b4edc087fa9a` | hub **v0.84.0 → v0.85.0** | -| felhom-agent | `6b5dade4dcc5` | agent **0.118.1 → 0.119.0** | +| **D-a** — merge the backup partition away (not resize) | `CONTEXT.md` S-5 | **R-165** (new) | +| **D-b** — desired/observed app state, own store | `CONTEXT.md` S-5 | **R-166** (new, `BLOCKED`) | +| **D-c** — storage monitoring + backup alerts | `CONTEXT.md` S-5 | **R-167** (new) | +| **D-d** — only DooPlex and Peti's box are protected | `CONTEXT.md` S-5 | `runbooks/target-selection.md`, this session — no row; the decision *is* the change | +| **Maintenance rule** (Part 3) | `STATUS.md` header **and** `CLAUDE.md` § End-of-session checklist | — | -## 2. The ask, and what it turned out to be +Recorded verbatim inside D-b, because it is the decision's binding constraint: *losing the state store +must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the +worst acceptable outcome is re-running a backup that already ran.* Its two "establish before speccing" +items are carried on R-166 as the reason that row is `BLOCKED` rather than `READY`. -*"Show the LAN and WireGuard IPs on the hosts page — it looks like it is not visible anywhere."* -Nearly exact, and the two halves were very different jobs: +**Deliverable 5 asks for "the five decisions".** Part 2 defines four (D-a … D-d); the fifth deliverable +line is the Part-3 maintenance rule, and it is in the table above. Nothing else in the task reads as a +fifth decision — flagged rather than invented. -- **WireGuard** — the hub already had it (`wg_peers.assigned_ip`, `GetWGPeerForHost`, **no UI - consumer**). It was visible in exactly one place, `/offsite`'s peer table keyed by pubkey, so you - could go peer→host and never host→peer. -- **LAN** — **not renderable at all.** `HostReport`/`HostMetrics` carried no address of any kind, so - this needed a cross-repo agent release, not an edit. Confirmed against the struct and against a - live box before proposing anything. +**D-a's two conditions are recorded as conditions, not commentary:** it changes the disk layout so it +must land **before any external install**, and it removes a wall that currently fails safely so +**R-167 ships in the same step, never after**. R-165 restates both; R-167 names R-165 as the thing it +gates. -Operator chose: both together as one arc, with the `addresses[]` (iface + cidr) shape. +**None of D-a, D-b or D-c is implemented.** No controller, agent, installer or hub file was opened for +editing. -## 3. Files (hub half) +## 3. R-163 re-framed, and R-156's papra referral resolved -- `hub/internal/web/hosts.go` — `parseHostAddresses`, `hostAddressView`, `hostNetworkView`, - `hostNetwork`, `minAgentForAddresses`, `hostDetailData` +1 key -- `hub/internal/web/templates/host_detail_body.html` — the Network card -- `hub/internal/web/hosts_network_test.go` (**new**) -- `hub/internal/api/testdata/host-report.golden.json` — the cross-repo contract, moved in lockstep -- `hub/CHANGELOG.md`, `REUSE.md`, `manifests/hub.yaml` -- `documentation/architecture/00-capability-map.md` (new row), - `documentation/audits/host-addresses-visible-2026-07-31.md` (new) +**R-163 is re-framed, not closed** — as instructed. State went `WAITING-ON-OPERATOR — the ratio is a +tier-sizing ruling` → `RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio`; "Blocked on" went +from `the operator's sizing decision` to a pointer at R-165; owner `operator` → `CC`. The cell now says +the sizing **question is withdrawn rather than answered**, that the row survives as the record of the +constraint until the merge lands, and that the original finding follows unchanged. The intake ranking +(item 4) was updated with it, and records that **R-165 inherits R-163's rank and is the highest-ranked +item that must land before any external install**. -## 4. Commits pushed to `main` +**R-156's papra referral is resolved.** The referral existed because moving a mount relocates live data +out from under a running app; with papra deployed nowhere there is nothing to strand, so the cheaper +leg — the template mounts `/app/app-data` — is takeable without waiting on upstream. -| Repo | Commit | What | -|---|---|---| -| felhom-agent | `14642e3` | v0.119.0 — the host report carries the box's addresses | -| felhom-agent | `4663df7` | REPORT | -| felhom.eu | `e07d90f` | hub v0.85.0 — Network card | -| felhom.eu | `37f7ff6` | manifests: hub 0.84.0 → 0.85.0 | +**The provenance is recorded with the claim, because it decides the row.** The evidence is +`docker ps -a` on **demo-hp's guest 9201** returning empty, **supplied with the task**; this session +**did not re-measure** — it is documentation-only and every box was fenced. The recorded scope is +honest about its edge: it covers the one guest papra was convicted on in Campaign 10, and **no other +customer's guest was enumerated**, so the row instructs the task that edits the template to re-check +first. Next action on the row is the catalog edit plus `catalog_gates.py`, explicitly not done here. -## 5. Tests + red-proofs (hub half) +## 4. `target-selection.md` per D-d -Suite **559 → 566** (+7). `go build ./... && go vet ./... && go test ./...` **rc=0**, run separately -from the commit per standing rule 1. +The rule at the top is now D-d: **two protected machines, everything else disposable**, with the +correction stated as a correction — the earlier caution was costing sessions and pushing drills onto +DooPlex. The tier table's Tier 2 row is DooPlex + Peti's cluster "and, by D-d, nothing else". -| Test | Asserts | Result | -|---|---|---| -| `TestNetwork_A_LANAddressRendered` | the LAN address + its interface reach the page | PASS | -| `TestNetwork_B_WireGuardAllocatedAndConfirmed` | WG shown once, marked confirmed, not duplicated as a plain row | PASS | -| `TestNetwork_C_AllocatedButBoxDoesNotHoldIt` | the drift case renders `not confirmed by the box` | PASS | -| `TestNetwork_D_OldAgentSaysUnknownNotNone` | an old agent renders UNKNOWN, never "no addresses" | PASS | -| `TestNetwork_E_NoPeerAllocated` | no peer → says so; the box's wg address is not silently dropped | PASS | -| `TestNetwork_F_NoReportYet` | never-reported host says "waiting", not "no addresses" | PASS | -| `TestParseHostAddresses_RealWire` | parses the real wire; empty/malformed degrade without panic | PASS | +**Two consequences the decision did not name, both handled visibly rather than silently:** -**The report fixture is the REAL wire** — the `addresses` block copied out of -`felhom-agent --selftest=hub` on demo-felhom running 0.119.0. A hand-written shape would have proved -only that the parser matches my own idea of the format. +- **`ep0` + the Hetzner Storage Boxes.** D-d's protected list has two machines and ep0 is not one, so + the page no longer calls it Tier 2. It is **not** thereby scratch: it holds the PBS-DR datastore and + the restic copy of a real customer's data — the only off-premises copy that exists. Read the narrow + way (not protected, but not wipeable), using the page's own *fences-name-acts* rule, and **flagged + in the page for the operator to confirm explicitly.** +- **The shared "do not re-point either backup target" fence** on the two demo boxes was **downgraded + from a prohibition to a stated cost**, because D-d makes both boxes freely reinstallable, which + spends that reference configuration just as thoroughly — keeping the fence would have left the page + self-contradicting. The reason survives: know you are spending the regression reference, and put the + box back. -| # | Mutation | Observed | Reverted | -|---|---|---|---| -| A | `hostDetailData` ships an empty `hostNetworkView{}` (the inert-seam shape) | `FAIL … the LAN address is not on the host page — the feature shows nothing` | yes | -| B | `WGConfirmed` set unconditionally from the allocation | `FAIL … an allocated-but-unheld peer renders as healthy — the drift is invisible` | yes | -| D | delete the `AgentTooOld` branch | `FAIL … an old agent's silence is not explained` | yes | +Also corrected while in the file: the *fences-name-acts* example cited the fence this edit removed, and +demo-hp's access line asserted "no baked SSH key" — which **R-129** records as measured false on +2026-07-31. It now points at R-129 instead of sending the next session to the hub vault for a +credential it may not need. -**Red-proof A took two attempts, and the first one proved nothing.** The initial mutation -(`_ = network` inside a map literal) did not compile, so the "failure" was a build error, not a test -result. Recorded because a red-proof that fails to build is indistinguishable from one that lands if -you only read the exit code. +## 5. The maintenance rule -## 6. Deployed +In two places, as specified: the `STATUS.md` header block, and a new **`## End-of-session checklist`** +in `CLAUDE.md` — which also gathers the couplings that were previously scattered (CHANGELOG + REPORT, +REUSE, the capability map's own end-of-session line, S-1's architecture coupling) and closes with *a +finding goes in `OPEN-ITEMS.md` first, never only in a report, an audit or `STATUS.md`*. -Image `gitea.dooplex.hu/admin/felhom-hub:0.85.0` built on DooPlex and pushed; `manifests/hub.yaml` -bumped in git; ArgoCD hard-refreshed and **deliberately synced** (never `kubectl set image`). +`CONTEXT.md` gained a header block stating why it and `STATUS.md` are separate — same subjects, +different readers, and `STATUS.md` holds nothing of its own. `STATUS.md` says the same from its side. -- ArgoCD app `felhom`: **Synced / Healthy** -- `deploy/hub` image: **`gitea.dooplex.hu/admin/felhom-hub:0.85.0`** -- Rollout: `deployment "hub" successfully rolled out` (rc=0) -- Startup log: `2026/07/31 08:50:57 [INFO] felhom-hub 0.85.0 starting` +## 6. What could not be translated into plain language -**One sequencing note worth keeping:** the first sync fired before ArgoCD had re-read git, so it -reported `Synced/Healthy` while still running 0.84.0 — a green pair that meant nothing. The fix is to -wait until `.status.sync.revision` equals the pushed commit **before** trusting the sync, which the -final run did. +Asked for explicitly, because an untranslatable row usually means the row itself is unclear. -## 7. Live validation +- **R-29** — *"gates are enforced nowhere"*. The class is stateable ("we have checks nobody runs"), but + its instances are four differently-broken scripts across two repos with no shared consequence, so + every plain sentence either says nothing or misstates one instance. **R-161 is its readable + fragment**, which is why R-161 is on the page and R-29 is not. +- **R-123 / R-125** — process findings about how the register and how tests are written. Real, and they + belong on the register; there is no customer-visible symptom to lead with, so they have no honest + first sentence for this page. They are not "broken" in the operator's sense. +- **R-133 (the plaintext break-glass credential)** — translatable, and left off only for space. It is + the strongest candidate for the next update if something else closes. +- **R-115 vs R-110** — separate rows, one plain-language paragraph. Merged into a single "Waiting on + you" bullet carrying both pointers, because two adjacent bullets about publishing read as one item + the operator has already half-decided. -**Method: endpoint-level** (no browser on DooPlex), against the deployed hub, cross-checked -against `ip addr` read directly over SSH on both boxes. +**A register defect found while reading, filed here because the fix is not mine to guess: `R-133` is +used TWICE** — `OPEN-ITEMS.md:80` (duplicate `domain` values accepted by the hub) and `:86` (the +plaintext break-glass credential). Two different findings, one ID, both `READY`. One needs renumbering, +and which one is the operator's call since both are cited from elsewhere (`CONTEXT.md` S-4 cites the +credential one). -| Host | Agent | Network card, live | Matches ground truth? | -|---|---|---|---| -| `demo-felhom-8363b5` | 0.119.0 | WG **10.77.0.2 confirmed**; `vmbr0 192.168.0.162/24`, `tailscale0 100.70.170.35/32` + ULA | **yes, exactly** | -| `demo-hp-bb76ea` | 0.119.0 | WG **10.77.0.3 confirmed**; `vmbr0 192.168.0.87/24`, `tailscale0 100.76.96.79/32` + ULA | **yes, exactly** | -| `drill-r50-0a4f9a` | 0.113.0 | WG 10.77.0.4 **allocated**; *"agent (0.113.0) does not report its addresses — they are unknown, not absent. Needs 0.119.0 or newer."* | correct UNKNOWN rendering | -| `sess-f-2670b5` | old | same UNKNOWN rendering | correct | +## 7. Files changed -Correctly absent from every card: `lo`, `vmbr9`'s `169.254.253.1` island literal, every `fe80::`. +| File | Change | +|---|---| +| `STATUS.md` | **new** — the operator page | +| `CONTEXT.md` | S-5 (D-a … D-d); header note on the `STATUS.md` separation | +| `documentation/backlog/OPEN-ITEMS.md` | R-165/166/167 filed; R-163 re-framed (state, blocked-on, owner, ranking); R-156's referral resolved with its provenance | +| `documentation/runbooks/target-selection.md` | D-d rule; tier table; ep0; the demo-box fence; two stale lines | +| `CLAUDE.md` | new `## End-of-session checklist`, carrying the `STATUS.md` maintenance rule | -**A fifth host 404'd and it was not a bug.** `sess-g-e687c3` appeared in a host list I read 30 -minutes earlier and returned 404 when I fetched its page; the concurrent session had deleted it as -part of its scratch-customer cleanup. Chased to ground before reporting rather than filed as a defect. - -## 8. Not yet validated / not covered - -- **A box with two LAN bridges** — handled by design (one row per interface) and unit-tested; no such - box exists to observe. -- **A real WireGuard drift** — the `not confirmed` branch is unit-tested and red-proofed but never - seen in production, because no live peer is currently unapplied. -- **IPv4-only / IPv6-only hosts** — every box here is dual-stacked via tailscale's ULA. - -## 9. Teardown - -This run provisioned nothing — no guest, no VM, no scratch customer, no external resource. The only -mutations are two in-place agent upgrades (previous binaries retained as `.bak-0.118.1` on both -boxes), the Day-0 manifest agent vouch, and the hub image bump. - -## 10. Backlog - -- **Opened: none.** **Closed: none.** -- **R-115 / R-121 touched, not closed.** This release walked all three legs deliberately — - deployed, **published** (round-trip verified) and **vouched** (read back) — so no new drift - instance was created. The finding those rows record is that nothing *notices* when the legs - diverge, which is unchanged. One small contribution: the Network card now surfaces - "this host's agent is older than 0.119.0" per host, which is a per-field hint, not the - installed-vs-vouched checker R-121 asks for. -- **Concurrent session:** every path was staged explicitly; no foreign WIP was swept into any commit. +Nothing was built, deployed, published or touched on any host. Every claim about the register above is +a claim about pushed source in this repo at the commit below. diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..7cb6b09 --- /dev/null +++ b/STATUS.md @@ -0,0 +1,66 @@ +# STATUS — what works, what's broken, what's next + +**Updated 2026-08-02.** + +> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this +> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**, +> which is technical state written for Claude Code — keep the two separate. **Maintenance:** update +> at the end of every session in which something shipped, broke, or was decided. One screen; cut +> items rather than extend it. + +## What works right now + +A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer, +who sets their own password. They install apps from a catalogue of fifty-three, share files over the +home network, and open apps from a launcher or a shared link. Backups run on their own to three +places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can +restore files and app data from the drive alone. Proven end to end on real hardware. + +## What's broken + +**After a power cut, an app can stay switched off — and nothing says so.** The machine looks for apps +that didn't come back, but looks too early and never again. In one case it decides the customer +switched it off deliberately, so it isn't even counted as down. *(R-157)* + +**The off-site copy can be erased by the machine that made it** — the credential that writes it can +also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy. +*(R-95, R-87)* + +**Three apps out of fifty-three kept their data where backups never looked.** They reported healthy; +the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is +installed nowhere. *(R-156)* + +**Local backups get 20 GB while apps get 50 GB.** An app that outgrows the smaller space stops being +backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is +lost: the last good copy is kept intact. *(R-163)* + +**When that happens, only one page says so** — no email, no alert. The page that answers "is this app +backed up?" is the one that stays silent. *(R-158)* + +**The check that catches this needs a person to remember it.** One command, run by hand; nothing +refuses a change that skipped it. *(R-161)* + +## What we're working on + +- **Now:** the last app whose data was never saved; today's decisions written down. +- **Next:** merging the small backup partition into the large one, with the drive-filling warning and + the backup-failure alert in the same step. +- **After:** rebuilding how the machine records whether an app is meant to be running. + +## Waiting on you + +- **How a new version reaches a machine.** Pushing the installer publishes it — half a minute later + every new machine downloads it, with no staging and no way back but another push. And publishing is + a step we remember rather than one the release performs, forgotten twice: a fix can be live here + and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to + settle both. *(R-110, R-115)* +- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a + session log; nothing suggests anyone else saw it. *(R-132)* + +## Changed since last update + +- **2026-08-02** — Decided: the 20 GB backup partition goes away and shares space with app data. That + changes the disk layout, so it happens before any machine is installed outside the house. +- **2026-08-02** — Decided: only this machine and the tester's box are protected; every other box, + both demo boxes included, may be broken or reinstalled freely. +- **2026-08-02** — Two of the three apps that never saved their data are fixed; this page created. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 247ce5d..fabc9e0 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -84,15 +84,18 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-137** | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with ""` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC | | **R-138** | **A shared-zone `cf_api_token` is a zone-wide DNS-write capability on a customer's box** — written 0600 to `/opt/docker/stacks/traefik/.env` (`controller/internal/infra/infra.go:123`) | READY (S) | — | Today each box holds a token for a zone nobody else uses, so the blast radius is one customer. Under a shared customer zone, one compromised tester box could repoint every other tester's DNS. The ACME path is already switchable — an empty token selects HTTP-01 (`traefik.yml.tmpl`) — so the fix is policy plus a guard that refuses to hand a shared-zone customer a zone-scoped token. Same audit §5.2 | CC | | **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC | -| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. Next action is papra's referral only. See R-161 (nothing runs the gate) and R-159/R-160 | CC | +| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **READY (S)** — the class is detected; papra itself is open | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. **THE REFERRAL IS RESOLVED, 2026-08-02 — papra is deployed NOWHERE, so the template fix strands nothing and can be applied.** The referral existed because changing where the volume mounts moves live data: an installed papra writes `db.sqlite` into the container's writable layer, and a remount relocates the path out from under it. With no instance deployed there is no live data to move, so the cheaper leg — **the template mounts `/app/app-data`** — is takeable directly, without waiting on upstream to adopt `/app/data`. **Provenance, stated because it decides the row:** the observation is `docker ps -a` on demo-hp's **guest 9201** returning empty, supplied with the 2026-08-02 task; **this session did not re-measure** (documentation-only, every box fenced). **Scope of that evidence, honestly:** it covers guest 9201 — the guest papra was convicted on in Campaign 10 — and **no other customer's guest was enumerated**, so a re-check belongs in the task that edits the template, before it edits it. **Next action: apply the template fix in `app-catalog-felhom.eu` and re-run `scripts/catalog_gates.py`** (deliberately not done here — that repo was out of scope for this task). See R-161 (nothing runs the gate automatically) and R-159/R-160 | CC | | **R-157** | **`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms, and a settle-condition fix closes only one.** **(A)** the sweep runs ~5 s after controller start (`cmd/controller/main.go:236`) while docker is still restoring, sees "no boot-orphaned apps", and never re-checks — **intermittent, 3 of 6 hard resets**; the predicate itself is sound (`bootrecon.go:100`, and `IsDownState` includes `StateDegraded`, `manager.go:55`). **(B)** an interruption that lands mid-compose leaves the stack with **zero containers**, which is exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp check reported **`0 currently down`** while a `deployed: true` app was not running: **silent on every channel**. | **READY (M)** | — | **Two fixes, not one.** A: a settle condition before the single observation, or one re-evaluation at the end of `deadAppBootGrace`, still bounded to the same attempts. B: needs a signature that distinguishes "customer stopped this" from "interrupted between containers" — B is F-CRIT-1's shape (`CLAUDE.md` false-invariant #4) reached via hard reset rather than quiesce. Test the **consequence** (app running again before the grace expires), not the mechanism. Evidence: `audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md` §5, `tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md` | CC | | **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC | | **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `_` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC | | **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC | | **R-161** | **The volume-persistence gate is enforced by CONVENTION, not automatically.** The catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none exists). | **REDUCED SCOPE — open** (operator ruling 2026-08-02) | a second person touching templates | **RULED. Both obvious enforcement points were rejected for measured reasons.** *Controller-side at template load:* rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — **it would pass on the exact defect it exists to catch**; the property is decidable only at runtime. *CI:* rejected for now — neither repo has any, and there are no users yet. **SHIPPED instead** (`app-catalog-felhom.eu` `fd7747d`): `scripts/catalog_gates.py`, ONE entry point running all three gates, non-zero exit on any failure, **mandated in the catalog's `CLAUDE.md`** the way `site_gates.py` is. Rationale for the record: of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md — `site_gates.py` is run, R-29's three orphans are named nowhere and have stopped nothing. **What remains open is only the automatic half:** this is convention, run by a person, and that is sufficient while one person touches templates. Revisit when a second does | operator | | **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC | -| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | the operator's sizing decision | **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 | operator | +| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio** | — (the sizing question is answered; the work is **R-165**) | **RE-FRAMED, NOT CLOSED (operator decision D-a, 2026-08-02 — `CONTEXT.md` S-5).** The row asked *what ratio should `mp1` be?* and that question is **withdrawn rather than answered**: `mp1` is merged into `mp0` so local recovery units share the app-data area and the ceiling stops existing — a bigger number is the same wall further away. **This row stays open as the record of the constraint** (what `mp1` is for, what it gates, and the measured 2× DB-app unit size) **until the merge lands**, because until then every consequence below is still live on every box. **The work is R-165; the warning that must ship with it is R-167.** Original finding, unchanged, follows. **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 (widened to R-167) | CC | | **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC | +| **R-165** | **Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.** Operator decision **D-a**, 2026-08-02 (`CONTEXT.md` S-5). Local recovery units share the app-data area instead of holding their own fixed ceiling, so the wall R-163 describes is removed rather than moved further away. Guest 9201 on demo-hp ships `mp0 50G / mp1 20G` today | **READY (M) — NEW 2026-08-02** | — | **Two conditions travel WITH the decision and are not optional.** **(1) Before any external install.** It changes the **disk layout**, so it is a fresh-install shape while there are no external boxes and a per-box migration after — and the decision's cheapness is entirely a function of that ordering. **(2) It removes a wall that currently fails safely**, so **R-167** (D-c: fill warning + failure alert) lands in the same step, never after: today an app that outgrows `mp1` is refused per app with the last good unit preserved byte-identical (R-158's measurement), and after the merge the same overflow consumes the space the app itself is using. Touches the installer/agent guest shape (`--sysdata-grow` defaults to **0** and is not derived from the physical drive, `felhom-agent/cmd/felhom-agent/main.go:178`) and the golden. **Does NOT close R-163** — that row is the record of the constraint and stays open until this lands | CC | +| **R-166** | **App state gets a desired/observed model with its own store.** Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5). Three kinds of information, deliberately separated: **desired** (running, or stopped because the customer said so) stays in `app.yaml`; **in-flight operations** (a backup started and never reported finishing) are **persisted** — written before the operation and after it — so an interruption is visible after a reboot; **observed** (unhealthy, restarting, drive gone) is **not** persisted and is rebuilt by looking. The last two live in a store **separate from `app.yaml`**, so frequent writes never touch a file holding deploy settings and secrets | **BLOCKED — NEW 2026-08-02** | two facts to establish before it is specced | **NOT a spec yet, deliberately.** **Establish first:** (a) whether the crash-safe journal already in the backup code covers the in-flight case — *it may already exist and merely be unwired, which has been the answer four times in this project* (`CLAUDE.md` "seam built but never wired"); (b) whether the existing SQLite store is reachable from this code path. **Binding safety rule, verbatim from the decision:** *losing the state store must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the worst acceptable outcome is re-running a backup that already ran.* The controller must also check **every container of an app**, so a partly-dead app cannot read as healthy. **This is the durable fix for R-157 mechanism B** (zero containers reading as a deliberate stop) and for F-CRIT-1's class; R-157 keeps its own row because it wants a bounded fix now, not this rework | CC | +| **R-167** | **Storage monitoring and backup alerts.** Operator decision **D-c**, 2026-08-02 (`CONTEXT.md` S-5). Two pieces: a warning to the **customer** as a drive fills, **before** anything fails; and a failure alert to the **hub/operator** when any backup fails for **any** reason, lack of space included | **READY (M) — NEW 2026-08-02** | — | **Prerequisite for R-165** — D-a removes a wall that currently fails safely, so this ships in the same step, not after. **Subsumes and widens R-158**, which is the same alert one tier down (`NotifyBackupFailed` exists and the hub allowlists `backup_failed`; only the off-box/NAS leg calls it, `cmd/controller/main.go:659`). R-158 stays as the named seam and its proposed `unitNotify` shape is the concrete first leg — **do not file a second row for the same wire.** The customer half is new: nothing today warns before a fill, and R-158's measurement is that when it happens exactly one page says so | CC | ## Why the TOP READY rows rank this way @@ -132,7 +135,10 @@ unrelated finding, which is why the sweep's proposals were renumbered to R-159 ran the gate; it now has a mandated entry point, so R-156's residue is the larger remaining item.)* 4. **R-163** — a real ceiling that silently caps local backup once an app outgrows `mp1`, and it gates Tier-2 and Tier-3 as well. Ranked below the above only because **overflow itself is safe today** — - it refuses per app and preserves the last good unit byte-identical. Waiting on the operator's ratio. + it refuses per app and preserves the last good unit byte-identical. **RE-FRAMED 2026-08-02:** no + longer waiting on a ratio — decision **D-a** merges `mp1` away, so the row is now the record of the + constraint and the work moves to **R-165** (with **R-167** shipping in the same step). R-165 inherits + this rank; it is the highest-ranked item that must land **before any external install**. 5. **R-158** — the gap that makes R-163 dangerous: cross the size line and **one page** tells you. On its own it is a notification gap, not a silent failure, which is why it sits here and not higher. 6. **R-164** — blocked on a predicate, no customer impact today; it only becomes urgent if the unit diff --git a/documentation/runbooks/target-selection.md b/documentation/runbooks/target-selection.md index 77147d1..20672ec 100644 --- a/documentation/runbooks/target-selection.md +++ b/documentation/runbooks/target-selection.md @@ -2,28 +2,48 @@ > Read before picking a machine for a drill, a destructive test, or a throwaway VM. It answers one > question: *what is safe to lose.* Reachability is in `CLAUDE.md`; hardware in `operations/nodes.md`. -> Added 2026-07-30. +> Added 2026-07-30. **Revised 2026-08-02 by operator decision D-d** (`CONTEXT.md` S-5). ## The rule -> **Start at Tier 0 and work down only if Tier 0 genuinely cannot host the work.** -> **A Tier 2 machine is used only when a task says so explicitly** — never by inference from what was +> **Two machines are protected: `DooPlex` and Peti's box. Everything else is disposable.** +> Operator decision **D-d**, 2026-08-02. DooPlex because it holds Gitea, the hub, the backups and the +> registry — everything else rebuilds from it, and it rebuilds from nothing. Peti's box because there +> is a real person behind it. **Every other box, both demo boxes included, may be broken or +> reinstalled freely.** +> +> **This is a correction, not a relaxation.** The earlier posture was costing whole sessions to +> caution and pushing drills onto DooPlex — the one machine that should never host them. If you are +> weighing whether a demo box can take a destructive test: it can. **Reach for a Tier 0 box first and +> do not ask.** +> +> Still true: **start at Tier 0 and work down only if Tier 0 genuinely cannot host the work**, and a +> protected machine is used only when a task says so explicitly — never by inference from what was > not forbidden. An absent fence is not permission. If no tier fits, **stop and ask.** -Fences name **acts**, not machines. "Do not re-target demo-hp's backup target" and "do not use demo-hp -to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine +Fences name **acts**, not machines. "Do not destroy demo-hp's `drill-r50` fixture" and "do not use +demo-hp to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine prohibition as covering the act it names and nothing more. | Tier | Meaning | Machines | |---|---|---| | **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) | | **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests | -| **2 — precious. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **ep0** + the Hetzner Storage Boxes | +| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** | **DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn. Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the thing under test, the source of truth for it, and the backups, at once. +**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected +list names two machines and ep0 is not one of them. **That does not make them scratch, and the +difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and +the restic copy of a real customer's data**, which is the only off-premises copy that exists, so +*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would +destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target. +**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has +read it the narrow way (not protected, but not wipeable) rather than assume the broad one. + **Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads. @@ -38,20 +58,24 @@ felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat **`/mnt/nvme-1tb`, at its root** (a subdirectory fails the agent's `exactMount` check → storage reads `disconnected` forever). **That warning is about one storage, not the box.** `/mnt/nvme-1tb` is also the `felhom-backup` target and the enrolled user-data drive, so remove scratch storages when done. -- **Forbidden:** do not re-target its backup configuration (below). Do not destroy or unblock - **`drill-r50` (VM 300)** — the only drift fixture (R-93). No baked SSH key; access is the hub-vaulted - G1 break-glass password (R-61). +- **Forbidden:** do not destroy or unblock **`drill-r50` (VM 300)** — the only drift fixture (R-93). + (Access: the docs say no baked SSH key and G1 break-glass, but a key authenticated on 2026-07-31 — + **R-129**, unresolved.) ### `demo-felhom` — N100 · **Tier 0** - **Freely:** create/destroy guests and scratch customers; reinstall the box. - **Care:** it carries the **PBS-DR / offsite tier** (demo-hp has none), so it is the Tier 0 box whose backup chain a test can actually disturb. Prefer demo-hp, per the 2026-07-25 ruling. -- **Forbidden:** do not re-target its backup configuration. -**Both Tier 0 boxes, one shared prohibition and its reason:** do not re-point either backup target. -They are the only two **correctly configured** boxes, so they are the regression path new installer -logic is measured against — losing that reference costs more than the box does. +**Both Tier 0 boxes — the shared backup-target fence is DOWNGRADED to a cost, 2026-08-02 (D-d).** It +read *"do not re-point either backup target"*, because these are the only two **correctly configured** +boxes and therefore the regression path new installer logic is measured against. D-d makes both boxes +freely breakable and reinstallable, which loses that reference just as thoroughly — so the fence was +inconsistent with the decision and is not kept as a prohibition. **What survives is the reason:** +re-pointing (or reinstalling) costs the reference configuration, so know that you are spending it and +put the box back. If both are spent at once there is no correctly-configured box left to compare +against. ### `DooPlex` — 192.168.0.180 · **Tier 2** @@ -71,7 +95,7 @@ still shares a device with its guest, so a drive failure is **offsite-only recov migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no enrolled host. No access route from DooPlex, and nothing here needs one. -### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2** +### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either** Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune