diff --git a/CONTEXT.md b/CONTEXT.md index 2dcf0a2..4caad30 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1019,7 +1019,9 @@ or retiring it is → **R-83**. over SSH** to a forced-command `felhom-peersync` script (pull rejected: weakens immediate revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev `felhom-hetzner` box, provisioned by the NEW runbook `documentation/runbooks/offsite-endpoint.md` - (WG on **443/udp**, subnet 10.77.0.0/24, PBS datastore `felhom-offsite` at `/srv/pbs-felhom`, + (WG on **443/udp**, subnet 10.77.0.0/24, PBS datastore `felhom-offsite` at `/mnt/pbs-datastore` + — a 100 GiB Hetzner Cloud Volume attached 2026-07-27 06:29:40 UTC; the pre-migration copy at + `/srv/pbs-felhom` is 13 G of dead weight on `/` awaiting R-91's go-ahead — nftables drop-all + wg0-only 8007, `ep0.felhom.eu` grey-cloud). Hub: `wg_endpoints`/`wg_peers` + /32 allocator + `/admin/wg/*` (global-key, pubkey-in-body) + `internal/wgsync` (FixedHostKey pin + **pinned-type HostKeyAlgorithms — live-caught bug**: multi-hostkey sshd presented ECDSA vs the diff --git a/documentation/PROMPT-TEMPLATE.md b/documentation/PROMPT-TEMPLATE.md index 7e1945e..f31acc8 100644 --- a/documentation/PROMPT-TEMPLATE.md +++ b/documentation/PROMPT-TEMPLATE.md @@ -326,9 +326,9 @@ Decisions made, architectural state, what's next. `/README.md` if architecture/features changed; authoritative architecture/feature docs in `felhom.eu/documentation/{architecture,controller}/`. Spikes/audits → `felhom.eu/documentation/audits/`. -### N.5 Capability map + backlog (end-of-session checklist — easy to forget) -If the task **changed what the platform can do** (new/removed capability, or a capability moved -status), update in the SAME session: +### N.5 The coupling rule — FOUR artifacts, same session (end-of-session checklist) +If the task **changed what the platform can do** (new/removed capability, a capability moved status, +or **what is open changed**), update **all four** in the SAME session: - **`architecture/00-capability-map.md`** — add/adjust the affected *scenario* row with the correct **status** (PROVEN-LIVE requires an `audits/`|`tests/` citation; else IMPLEMENTED/PARTIAL) and an evidence citation. A leg not exercised live → PARTIAL/IMPLEMENTED with a note saying which leg and @@ -337,6 +337,16 @@ status), update in the SAME session: collapse a now-shipped item to its one-liner + version, and honor the **coupling rule** (every roadmap item names the map row it flips; every map gap row points back by ID). Parked sub-features (e.g. an alternative transport, an agent-plane leg) go as a note under the owning arc's item. +- **the owning `architecture/*.md`** — ruled as **S-1** in `CONTEXT.md` (2026-07-26, R-81): any task + that changes an **architectural contract** (tiers, targets, cadences, trust boundaries) updates the + owning design doc in the same session. It was ruled but never written here, in the template CC + actually reads — so it bound nobody. Now it does. +- **`backlog/OPEN-ITEMS.md`** — the register, and the single source of truth for open work. A task + that changes what is open without touching it re-creates exactly the thread-loss the register was + built to solve: `REPORT.md` is overwritten every session, so nothing durable may live only there. + +**Report which `OPEN-ITEMS.md` rows the task opened, closed or re-ranked** (§15). Every row carries +an owner — a row nobody owns is how items got lost in the first place. ### N.6 Website version bump (if controller/hub version is shown on the site). diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index fe9be50..2244255 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -10,19 +10,19 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha |---|---|---|---|---|---| | **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — | | **R-88b** | `/backup/due` cannot say *unknown* — "read errored" and "never backed up" are byte-identical, so nil still bypasses the window gate | **READY #1** | — | Agent wire change: give unknown its own representation; compat rule both ways + MinAgent floor | CC | -| **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY #2** | — | Snapshot plan as the stopgap (row below), then split prune off-box or move to REST `--append-only` | CC | +| **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY #2** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC | | **R-94** | Hub hands out host-install `1.19.0`; `1.20.0` is what carries R-82's backup default | **READY #3** | — | Bump `configs.go:28`, and stop hand-syncing a version constant across repos | CC | | **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY #4** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC | | **R-87** | The restic tier is never restore-tested | **READY #5** | — | Design a controller-side test (no scratch-guest analogue transfers) | CC | -| — | Enable Hetzner Storage Box **snapshots** on `storage-box-pool-1` — `snapshot_plan=null`, 0/10 used, server-side so SFTP cannot delete them | WAITING-ON-OPERATOR | operator ruling | One console/API call; immediate immutability for the restic tier | operator | +| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC | | — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator | -| **R-90** | ep0: 3.8 GB, **no swap**, OOM'd 2026-07-27 killing PBS for ~15 min | **BLOCKED** | Hetzner CX33 availability | Rescale; or add a swapfile as an interim (needs no console) | operator | +| **R-90** | ep0 RAM headroom — **4 GiB swap added 2026-07-27** (persistent, verified without reboot); 3.8 GB RAM unchanged | **BLOCKED** (interim done) | Hetzner CX33 availability | Rescale when available; swap now covers Sunday's first GC | operator | | **R-91** | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC | | — | First-ever **GC** on `felhom-offsite` (armed today 13:11 UTC, never run) | WATCHING | schedule | **Sun 2026-08-02 04:30 UTC** — confirm it completes | CC | | — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC | | — | demo-felhom's next restore-test (84 h cadence, last 2026-07-27 06:38 UTC) | WATCHING | schedule | ~2026-07-30 18:38 UTC | CC | +| **R-97** | ~~Whole-guest backup tier had no hub signal; quiesce blamed the apps~~ | **SHIPPED** (controller v0.177.0 + hub v0.78.0, 2026-07-27) | — | Operator-only events proven end-to-end; app suppression is cycle-keyed with a 180 s expiring grace | — | | **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC | -| **R-96** | Two standing rules agreed in chat, never committed | READY (XS) | — | Add both beside `CONTEXT.md` S-1/S-2 | CC | | **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC | | **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC | @@ -32,7 +32,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha that an unknown still fires the safety valve, so a read failure can take **one** out-of-window quiesce. Bounded now, but it is the fourth appearance of this class and the only one still open. 2. **R-95** — the largest *data* exposure: the tier holding the customer's documents and photos is the - one whose credential can delete, and the mitigation is a console click nobody has made. + one whose credential can delete. The snapshot mitigation is now **armed** (daily 00:00, keep 7), + but it has taken zero snapshots so far and it does not touch the root cause — the box can still + `forget --prune` its own repo. 3. **R-94** — a one-line constant, but until it moves every hub-driven install gets the pre-R-82 backup default. Cheapest high-consequence fix on the list. 4. **R-86** — an operator ruling already exists; it only waits on knowing what load ep0 can take. diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 42fa345..386222b 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -124,7 +124,7 @@ Each attempt runs the **full quiesce cycle**, so every customer app stack is STO Self-resolves the moment the target answers (the storage read succeeds, sees the archive, tier stops being due) — which is why it can hide indefinitely: it needs an offsite outage to appear at all. **PHASE-0 ROOT CAUSE, established at source 2026-07-27 — it is AGENT-side, case (a).** The storage read **errored** (`could not read the backup storage for the due-check … err=…` at 09:02:57/09:07:58/09:12:57 CEST), so this was never an empty-success. The failure is a **type boundary**: `newestArchiveOn` (`localapi/server.go:1095-1111`) documents *"Errors and unsupported services degrade to unknown, never to 'no backup'"* — but its `(time.Time, bool)` signature **cannot represent unknown**, so an error and a genuinely-empty storage both collapse to `(zero, false)`, and `handleBackupDue` (`server.go:934-941`) then emits a POSITIVE claim: `Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil`. The fail-safe that *does* exist — `targetStoragePresent`'s "a storage-view error must never be read as 'not there'" (`server.go:1131-1151`) — answers a different question (does the storage exist) and behaved correctly. **Decisive for scoping: the errored path and the genuine-never path are BYTE-IDENTICAL on the wire** — same `Due`, same `Reason` string, same nil `AgeSecs` — so the controller has nothing to discriminate on and Part 2 CANNOT be done controller-side. **Two further P0 findings:** the agent restarted **4× on 2026-07-27** (07:36:39, 07:54:06, 08:50:16, 11:31:52 CEST) — all deliberate (`NRestarts=0`, `Restart=on-failure`, `Result=success`), zero self-update — so the trigger is armed by ordinary operator/config work far more often than "only when ep0 is down"; and **the loop alerted NOBODY** — zero `backup_failed` events despite the hub allowlist carrying that type, because **`internal/quiesce` does not import `internal/notify` at all**. Its only trace was `07:13:27 info app_start_failed "Telepített alkalmazás nem fut: BookStack"` — a customer-tier, Hungarian, info-severity SYMPTOM of the third cycle catching BookStack mid-restart. **The whole-guest backup tier R-82 built has no failure signal to the hub → its own item.** **Shape:** distinguish *storage unreachable* from *storage readable and empty*. Unreachable is UNKNOWN — defer the due-verdict rather than resolving it either way, exactly as R-81 made the hub do with a missing report. Only a target that is reachable AND has no archive is genuinely due. **Fix the window bypass in the same slice:** `AgeSecs == nil` must stop meaning "run now regardless of the hour". Either the agent distinguishes *never backed up* from *cannot tell* in what it reports, or `scheduledRunAllowed` gates on the former only — otherwise any future nil-age path re-opens the same hole. Note this does NOT weaken R-84's fail-safe intent: a tier whose storage is merely slow or briefly unreadable should still err toward backing up — it is specifically the **cold-store + unreachable** pair that must defer, because there the fallback has no information at all, only an empty default that looks like a fact. | | R-89 | **Retention is a COMMERCIAL attribute — it belongs to the hub, not to ep0 or a box** | M | idea — operator ruling 2026-07-27, first increment SHIPPED same day | **Ruling (2026-07-27):** retention is a per-customer *commercial* attribute (a paid tier may buy longer retention), so the **hub owns the policy** and ep0 merely executes it. Execution stays **server-side**: a reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the last-known policy running rather than silently stopping retention. **Increment 1 SHIPPED 2026-07-27** (`runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`): boxes no longer attempt prune (`keep_last: 0` → `allowPBSPrune=false`, config only — no code, no grant), and per-namespace prune jobs run on ep0 daily 03:30 UTC (`keep-last 2`), dry-run gated and verified `TASK OK`. This also closed a live defect — **every** demo-hp PBS backup since the tier was created on 07-26 had reported `job errors` while the data landed correctly, because `DatastoreBackup` grants `Datastore.Backup` but not `Datastore.Prune`; a tier that cries wolf on every success makes a genuine failure invisible, which is exactly what happened during the 07-27 migration window. **Remaining work:** retention becomes a per-customer attribute on the hub — a **policy** (keep-last/daily/weekly/monthly), NOT a bare number, since a paid tier will want "12 weekly + 6 monthly"; a reconciler converges it into the ep0 prune job in the same descriptor-and-converge shape as `pbs_dr`. **SECURITY PROPERTY TO PRESERVE — do not "fix" a future prune error by widening the grant:** box tokens stay **write-only** (`DatastoreBackup`), never `DatastorePowerUser`/`DatastoreAdmin`. A compromised box must not be able to delete its own offsite backups — that is the scenario offsite DR exists to survive. **PARALLEL QUESTION, unanswered and arguably the more urgent half: does the restic key on `storage-box-pool-1` (`u629488`) have DELETE rights?** If so the daily app-data offsite tier carries the identical exposure, and restic's **append-only** mode is the equivalent answer. **Rule once for both tiers.** Flips the capability-map row for offsite retention (currently: retention configured but unenforceable) | -| R-97 | **The whole-guest backup tier has NO failure signal to the hub — `internal/quiesce` never notifies** | S | idea — found by R-88 Phase 0, 2026-07-27 | On 2026-07-27 three whole-guest backups failed and three quiesce cycles stopped and restarted every customer app stack, and **not one `backup_failed` event reached the hub.** It is not the allowlist — the hub already carries `backup_failed` and `backup_completed` (they are emitted by the controller's *app-data* backup path). The cause is that **`internal/quiesce` does not import `internal/notify` at all**: the tier R-82 built has no route to the hub, so a whole-guest backup can fail indefinitely in silence. The loop's only trace was `app_start_failed` — **info** severity, **Hungarian**, on the **customer** channel — telling the customer BookStack was down (it had been caught mid-restart by the third cycle) without saying why, during an outage the system itself caused. So the one signal that did fire was both the wrong tier and the wrong story. **Shape:** emit `backup_failed`/`backup_completed` from `quiesceAndPollTiers` naming the TIER, operator-tier; and decide whether a quiesce-induced restart should suppress `app_start_failed` the way R-164's deliberate-stop filter does — an app the backup stopped on purpose is not a fault. R-88's breaker bounds the repetition but changes nothing about the silence | +| R-97 | **The whole-guest backup tier has NO failure signal to the hub — `internal/quiesce` never notifies** | S | **SHIPPED (controller v0.177.0 + hub v0.78.0, 2026-07-27)** — **R-97a:** `quiesce.TierNotifier`, a seam (not an import) wired by an init-only setter, edge-triggered on the R-88 breaker ARMING so a failing tier is reported once per run rather than once per retry; recovery rides `recordSuccess`'s existing bool. **NEW operator-only event types** `whole_guest_backup_failed`/`_recovered` — deliberately NOT `backup_failed`, which carries a customer Hungarian template AND sits in demo-felhom's live `enabled_events`, so reusing it would have emailed the CUSTOMER about a backup they cannot act on while it was still retrying. The recovery joins `recoveredPairedDownTypes` because its `info` severity would otherwise be dropped by `severityNotifies` — the operator would hear it break and never hear it heal. **The hub's operator cooldown was keyed `customerID:eventType` alone**, so one tier would have masked the other for an hour; now narrowly extended with a `tier` suffix taken from the event details, leaving every other event type unchanged. **R-97b:** a suppression window keyed to the quiesce CYCLE (not a state test — v0.164.0's `!= StateStopped` filter cannot see an app caught MID-RESTART, which is exactly how BookStack alarmed), consumed at the same single derivation point `classifyRunStates`. Grace = **180 s**, derived from the deploy flow's 120 s health timeout and Mealie's 60 s `start_period`; it **expires**, so an app that genuinely fails to come back still alarms. **PROVEN LIVE end-to-end with a control:** the new type POSTs 200 from inside guest 9201 while a bogus type 400s, and `notification_log` shows **1 operator row, 0 customer rows**. The quiesce→notify link itself is unit-proven only. | On 2026-07-27 three whole-guest backups failed and three quiesce cycles stopped and restarted every customer app stack, and **not one `backup_failed` event reached the hub.** It is not the allowlist — the hub already carries `backup_failed` and `backup_completed` (they are emitted by the controller's *app-data* backup path). The cause is that **`internal/quiesce` does not import `internal/notify` at all**: the tier R-82 built has no route to the hub, so a whole-guest backup can fail indefinitely in silence. The loop's only trace was `app_start_failed` — **info** severity, **Hungarian**, on the **customer** channel — telling the customer BookStack was down (it had been caught mid-restart by the third cycle) without saying why, during an outage the system itself caused. So the one signal that did fire was both the wrong tier and the wrong story. **Shape:** emit `backup_failed`/`backup_completed` from `quiesceAndPollTiers` naming the TIER, operator-tier; and decide whether a quiesce-induced restart should suppress `app_start_failed` the way R-164's deliberate-stop filter does — an app the backup stopped on purpose is not a fault. R-88's breaker bounds the repetition but changes nothing about the silence | | R-95 | **The restic offsite tier's credential CAN DELETE — R-89's "parallel question", now ANSWERED** | M | idea — established read-only 2026-07-27 | **The exposure closed on the weekly PBS tier is fully open on the daily restic tier**, which holds the customer's actual documents and photos and is the only tier that survives losing the box. Established without mutating anything: **(1) Identity** — a per-customer *subaccount* on `storage-box-pool-1` (box 611714, bx11, `u629488`): `u629488-sub1` home `felhom-demo-felhom`, `sub2` peti-felhom, `sub3` demo-hp, each labelled `felhom-customer`. Auth is an **SSH key stored ON THE BOX** (`…/felhom-controller-data/_data/data/offbox/ssh_key`, 0600, beside `repo_password` + a pinned `known_hosts`) — customer-side, not hub-side, so a compromised guest holds it. **(2) Read-write: YES** — the API reports **`readonly=False` on all three subaccounts**, and it is not merely latent: the controller runs `restic forget --group-by host,tags --keep-daily 7 --keep-weekly … --prune` **from the box** (`backup/offbox.go:984`, also `:1070`). Delete rights are exercised on every run. **(3) Append-only: NO, and not expressible** — the repo is built as `sftp:` (`offbox.go:482`); restic's append-only mode requires the **REST server** backend, which plain SFTP cannot provide. **(4) A zero-code mitigation exists and is unused:** the box type carries `snapshot_limit=10` and the API reports `snapshot_plan=null` with **0 snapshots** and `size_snapshots=0`. Hetzner Storage Box snapshots are taken **server-side, outside the SFTP namespace** — an SFTP subaccount cannot delete them — so they are a genuine immutability layer at no extra cost and with no code change. **Rule once for both tiers, per R-89.** Options, cheapest first: enable a snapshot plan (operator click, immediate); split backup-write from prune so pruning runs somewhere the box cannot reach; or move the repo to restic's REST server with `--append-only`. Flips the capability-map row for offsite immutability | | R-94 | **The hub hands out host-install `1.19.0` while `1.20.0` is what ships R-82's backup default** | XS | idea — found 2026-07-27 | `hub/internal/web/configs.go:28` pins `const hostInstallVersion = "1.19.0"`, and that constant renders the customer page's install command (`configs.go:487` `ScriptVersion`). `scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.20.0"` — the version R-82 shipped to default a fresh box to **local-daily + offsite-weekly**. So a new install driven from the hub still gets the **pre-R-82** default. This is the same constant hub `v0.73.2` existed to sync (→1.19.0); it went stale again one installer release later, which is the actual finding: **a hand-synced version constant in a second repo drifts every time the first one ships.** v0.73.2 itself is NOT an undeployed artifact — its content is inside the live 0.77.0. Related to R-82's "REMAINING GATE" note, but that gate is about the fleet flip; this is the hub simply pointing at the older script | | R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |