diff --git a/CONTEXT.md b/CONTEXT.md index 79d432f..34a847f 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -123,6 +123,33 @@ Two rulings, one shape: something became live because someone pushed, not becaus that bumps a version, before publishing — and a gate that fails on the normal path is one people learn to ignore. +**S-16 — a backup run NOTIFIES ONCE and RECORDS ALWAYS, and those are different things +(2026-08-03, R-182; controller v0.194.0 + hub v0.90.0/.1).** Measured: nine per-app capture failures +reached the hub, two were mailed, seven were dropped by a cooldown whose key carries no app +identifier — *before* `LogNotification`, so they left no row anywhere. + +- **The record:** `recovery_unit_capture_failed`, per app, unconditionally, now routed + **record-only** by the hub (`recordOnlyEvents`) — stored and logged every time, never competing + for an e-mail slot. +- **The notification:** `backup_run_failures`, ONE per run, listing every failed app with its leg + and reason plus the counts and free space. **A clean run emits nothing.** +- **A suppressed operator event now leaves a `suppressed` row** carrying the key that suppressed it — + for every operator type. *"We chose not to e-mail you"* and *"nothing happened"* must never look + identical; that is the whole finding, stated as a rule. +- **The periodic sweep gets a digest too, with NO `run_id`**, so it stays under the ordinary hourly + cooldown. Without it the sweep's failures would be recorded and never notified — a new silence + created while closing one. A real run's digest carries a unique `run_id` precisely so the cooldown + can never collapse a manual run into the nightly one. +- **Why the silence is safe:** the hub's deadline check raises `expected_backup_missed` from report + freshness, independently of any mail the box sends (`monitor/deadline.go:396,417`). **If that check + is ever weakened, this design loses its footing.** +- **Not taken, and why:** putting `app` in the cooldown key. It fixes the swallowing by producing one + mail per failing app — a dozen on a full disk. + +**ep0 was rescaled by the operator to a CX33 (2026-08-03): 4 vCPU, 8 GB RAM, measured on the box, and +the 4 GiB swapfile survived. The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only, so no disk +figure in any runbook needed correcting. That closed **R-90** and unblocked **R-86**. + **S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).** Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure diff --git a/REPORT.md b/REPORT.md index d699157..cca6d01 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,200 +1,188 @@ -# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed +# REPORT — R-182: one email per backup run, nothing dropped without a trace + ep0 rescaled (R-90, R-86) -**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**) -**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.** +**Date:** 2026-08-03 · **Repos:** `felhom-controller` **v0.194.0** (`88897a2`), `felhom.eu` hub +**v0.90.0 → v0.90.1** (`dd40f85`, `f21e7ca`) -## 1. Baselines — re-read on arrival, both matched §1 +## 1. Baselines — re-read on arrival -| Repo | @ arrival | Version | Result | +| Repo | @ arrival | Version | Matched §1? | |---|---|---|---| -| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` | -| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only | +| `felhom-controller` | `db0d4b129d3b` | v0.193.1 | **yes** | +| `felhom.eu` | `7dc1744eece3` | hub v0.89.0 | **yes** — my local clone was one commit behind origin (another session had deleted 24 stale `REPORT-*.md` files); pulled before starting | -## 2. Part 0 — the R-182 measurement, and it REVERSED the row +## 2. §3's safety property — confirmed at file:line -Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed -at the sending end. Measured at the **receiving end**, it is the opposite. +`hub/internal/monitor/deadline.go:396` raises `expected_backup_missed` and `:417` raises +`expected_dbdump_missed`, both from `assessBackupFreshness(reportJSON,…)` and `GetEventsByType` — the +box's **report freshness and stored events**, never from an email. So the hub notices a run that never +finished **independently of any mail the controller chooses to send**, and a digest's silence still +means "the run finished and found nothing wrong". **The whole design rests on this; weakening that +check re-opens a silent-failure path.** -Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db` -alone would have read stale data, the exact trap this project recorded before), freshness confirmed by -the newest `notification_log` row post-dating the session. +## 3. Files -**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.** +**Hub:** `internal/notify/dispatcher.go` (suppression row, `cooldownRunSuffix`, `recordOnlyEvents`, +`operatorOnlyEvents`) · `internal/notify/templates.go` (digest renderer, `trimRepeatedUsage`) · +`internal/api/handler.go` (allowlist) · `internal/store/store.go` (status doc) · two new test files · +`recovery_unit_dispatch_test.go` (updated, deliberately) · `hub/CHANGELOG.md` · `manifests/hub.yaml` -| time | apps refused (events in) | operator emails out | -|---|---|---| -| 06:40:03 | privatebin, opengist | **opengist only** | -| 08:59:46/47 | opengist, privatebin | **privatebin only** | -| 08:59:59 | privatebin, opengist | **none** | -| 09:03:00 | opengist | **none** | -| 09:07:06 | privatebin, opengist | **none** | +**Docs:** `CONTEXT.md` (S-16) · `STATUS.md` · `documentation/architecture/07-backup-architecture.md` +(the alerting contract, S-1) · `00-capability-map.md` · `backlog/{OPEN-ITEMS,ROADMAP}.md` · five +runbooks corrected · two audit/evidence documents annotated -**Cause, confirmed at source:** the operator cooldown key is -`customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded). -`RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key -holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the -next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any -channel** and cannot be audited afterwards. +## 4. Tests and red-proofs -This is **R-97a's failure mode in a second event type**; that row's own comment states it -(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was -written narrow on purpose; `recovery_unit_capture_failed` simply never opted in. - -**A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app, -HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the -operator was told about each app* — which is false. The gap between an accepted event and a sent -email is the whole of this row. - -**Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape. - -## 3. Probes - -| | Question | Method | Verdict | -|---|---|---|---| -| **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) | -| **P-B** | does Gitea serve `raw/tag//`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` | -| **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** | - -**Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in -~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone -(`git ls-remote --tags` → 0 at the time). - -## 4. §8.2's three channels — enumerated - -| Channel | Before | After | | -|---|---|---|---| -| 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs | -| 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below | -| 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission | - -**Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.** -There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so -no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"* -was impossible for them. Raised before building; operator ruled to pin them to **the agent version -being installed**, which the installer already resolves from the hub manifest and already sha-verifies. -That is strictly better than any installer-derived tag: binary and configs now come from one ref. - -**Channel 3 needed no change because the URL never carried a ref** — -`https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So -`felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template -change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose. - -## 5. The tag convention - -- **Shape:** `installer-v` in `felhom.eu` (prefixed so it cannot be read as a hub, - agent, controller or golden version); `v` in `felhom-agent` (that repo versions one thing). - **No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time, - and channel 1's ref lives only in the manifest. -- **Publish:** cut `installer-v`, bump the `--ref` in `webpage.yaml` (sidecar *and* - init container), commit, sync. -- **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**. - -## 6. Scenario A — proven by HTTP - -A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and -three sync periods were allowed to pass so "unchanged" means "had every chance to change": - -``` -website tree (main): .worktrees/6a82719… <- ADVANCED to the new commit -scripts tree (tag): .worktrees/bee6848… <- STAYED -sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555 -sha256 after push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555 -marker present at the served URL? 0 -https://felhom.eu/ -> HTTP 200 -``` - -Both halves of the split in one observation: the site still tracks `main`, the installer does not. - -## 7. Scenario B — publish and rollback, both directions - -| act | result | -|---|---| -| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** | -| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** | - -`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved -to `main`'s head — a **byte no-op**, verified by the served sha not changing. - -## 8. Files, commits, tags - -**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b` -(marker removed), plus the docs commit below. -`scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` · -`manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` · -`documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md` - -**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT). -`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** · -`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md` - -**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and -`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from; -`configs/` is byte-identical there and at `main`, so nothing depended on the choice). - -## 9. Tests and red-proofs - -| Check | Result | -|---|---| -| `felhom.eu` `repo_gates.py --fast` | all 5 gates OK | -| `felhom-agent` `go build ./... && go vet ./...` | OK | -| `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) | -| `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) | -| `agent_gates.py` (full) | both OK | - -**Red-proofs, each demonstrated failing then restored:** +`go build && go vet && go test ./...` in `hub/` → **17 packages ok, rc=0**. `repo_gates.py --fast` → +all 5 gates OK. | # | Mutation | Result | |---|---|---| -| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired | -| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it | -| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state | -| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 | -| F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 | +| D | the suppression row removed (back to the bare `return`) | **RED** — `want 1 'suppressed' operator row, got 0` | +| E | `cooldownRunSuffix` removed from the key | **RED** — `operator mails = 1, want 2`; the manual run's digest is swallowed | +| G | the digest removed from `operatorOnlyEvents` | **RED** — and it showed **a customer actually receiving the operator digest**, demonstrated rather than argued | +| A | (controller) all four `noteFailure` feeds removed | **RED** | +| I | (controller) the `main.go` seam commented out | **DID NOT FAIL FIRST TIME** — see below | -**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit* -`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also -retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was -this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API -token, and the only credential available (`~/.docker/config.json`) is a registry password that the API -rejects — so the controlled before/after replaced the log rather than an assumption standing in for it. +**Red-proof I is the one worth reading.** The AST test walked the backup package but not `main.go`, so +the seam could be disconnected with the whole suite green. Per §9.11 that is not a pass: **the test was +fixed and the mutation re-run**, at which point it failed correctly. The string `SetRunSummaryNotify` +is still present in the commented-out file, which is exactly why this is an AST walk. -## 10. No version bumps, nothing built +**One passing test was changed, deliberately.** +`TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer` required the OPERATOR to be emailed a per-app +capture failure — correct when that event was the only signal, wrong now that it is the record and the +digest is the notification. Its **customer-safety claim is unchanged** and is why the test still +exists; the operator assertion is inverted with the reasoning written in place, and it now asserts the +`recorded` row instead. R-158's guarantee **moved rather than weakened**. -`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file -touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's -no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or -golden was built. +## 5. The live run -## 11. Register +Demo-hp guest 9201, filled for real to **241 MB free / 100% used** (thin pool held **30.78 → 30.78**). +Triggered through `POST /api/debug/backup/dbdump` — the exact endpoint the debug UI button calls. + +**The hub's `notification_log`, quoted:** + +``` +11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "opengist" … +11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "privatebin" … +11:50:45 | backup_run_failures | operator | sent | | 2 of 2 apps failed to back up in this manual run: opengist, … +11:50:45 | backup_run_failures | customer | skipped | operator_only | 2 of 2 apps failed … +11:51:00 | disk_critical | operator | suppressed| operator cooldown 1h, key=demo-hp:disk_critical | … +11:51:00 | disk_critical | operator | sent | | … +``` + +Two failures → **both recorded**, **one** email naming **both**, customer refused with the reason +logged. Against the measured previous behaviour: two failures → one email naming one app, one leaving +no trace. + +**Scenario D proved itself on an event I had not planned.** `disk_critical` alarmed on two +filesystems; the second was collapsed by the cooldown, and that collapse is now **visible with its +key**. Yesterday it would have left nothing at all — which is the entire finding, demonstrated by +accident on a different event type. + +**Scenario E:** a second run in the same hour (11:51:28) produced a second digest. +**Scenario B:** after freeing the space, the run completed with 2 volume dumps and **no** digest line. + +## 6. The email as shipped + +``` +SUBJECT: [Felhom] 🔴 demo-hp: 2 of 2 apps failed to back up (manual run) + +Customer: demo-hp +Event: backup_run_failures +Severity: error +Time: 2026-08-03 13:52 CEST +Message: 2 of 2 apps failed to back up in this manual run: opengist, privatebin + +FAILED: 2 of 2 apps attempted in this manual run. + + opengist whole app (refused before any write) refused: backing up this app would leave the + filesystem below the reserve (reserve: 97% used + or 1.0 GiB free; the filesystem is already below + it, before this app's estimated 178.0 KB write) + privatebin whole app (refused before any write) refused: … estimated 2.5 KB write + +Filesystem: /mnt/sys_drive — 65.0/68.7 GB used (95%), 0.2 GB free + +Every failure above is also recorded individually in the notification log, +whether or not this mail was sent. + +Dashboard: https://hub.felhom.eu/customers/demo-hp +``` + +**v0.90.1 came from reading exactly this.** In the first rendering every app row also ended with +`— /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free` — the same clause the `Filesystem:` line +already carries. On two apps that is untidy; down a list of a dozen it is forty characters twelve +times, pushing the part that *differs* off a phone screen at 07:00. The reserve's message is unchanged +(it is right for a single-app alert); the digest trims the duplicate when rendering, and only an exact +`— :` suffix. + +## 7. §8.1's table, in practice + +The run refused **both** deployed apps on the reserve → both in the digest. **No** deliberate skips +occurred in the live run (no drive was disconnected or decommissioned on that box), so that row of the +table was exercised by test only — `TestRunSummary_DeliberateSkipsAreNotFailures`, which marks one app +disconnected and one decommissioned through the real `settings` store and asserts **no digest at all**. + +## 8. ep0 — measured, then corrected + +| | measured on the box | +|---|---| +| hostname | `felhom-hetzner` | +| RAM | **7757 MB (8 GB)** — was 3.8 | +| vCPU | **4** | +| swapfile | **survived the resize** — `/swapfile`, 4 GiB, `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`) | +| local disk | `/` 38 G, 58% used — **UNCHANGED**, as expected for a CPU/RAM resize | +| datastore volume | `/mnt/pbs-datastore` 98 G, 8% — unaffected | + +The swapfile was **checked rather than assumed**: a resize is a stop/start, so "the interim lever is +still in place" was an assumption until measured. + +**Corrected** (five places, three of which the spec's list did not name — found by searching): +`RUNBOOK-ep0-datastore-volume-2026-07-27.md:6` · `RUNBOOK-pbs-prune-serverside-2026-07-27.md:232,:310` +· `runbooks/offsite-endpoint.md:8,:34` · **`runbooks/target-selection.md:101`**. + +**Annotated, body untouched** (they record what was true when written, and that is their value): +`audits/SPIKE-connectivity-wireguard-2026-07-03.md:13` · +`tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md:142`. + +**No disk figure was touched anywhere** — the 40 GB local disk did not change. + +## 9. Register | ID | Outcome | |---|---| -| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed | -| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) | -| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described | -| **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs | -| **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released | +| **R-182** | **CLOSED — SHIPPED**, with the live evidence above | +| **R-90** | **CLOSED** — ep0 rescaled to a CX33; measured on the box, swapfile confirmed surviving | +| **R-86** | **READY, unblocked and re-ranked** — endpoint headroom no longer sets the restore-test cadence. Stated honestly: 8 GB is comfortable, not unbounded (the original OOM was a 14.46 GB restore), so the cadence should still be paced — just not by fear of the endpoint | -**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each; -all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0 -hits anywhere and remains free. +**No new IDs were minted**, so no grep was needed. R-184 remains the highest. -## 12. Observations — noticed, documented, NOT acted on +**`target-selection.md`'s ep0 protection question is STILL OPEN and deliberately untouched** — +*"D-d did not name ep0 either way. Confirm it explicitly."* That is the operator's to settle, not CC's. -1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs - either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails - closed where the mistake is made, needs no new credential). -2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`, - so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside - R-182 because it is what made that row take a day to get the right way round. -3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it - means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate. -4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a - network gate is ever added there, that workflow needs the same change the agent's just got. +## 10. Deployed -## 13. Teardown +- Controller **0.194.0** on demo-hp guest 9201 — `Up (healthy)`. +- Hub **0.90.1** — ArgoCD `Synced` / `Healthy`, image `gitea.dooplex.hu/admin/felhom-hub:0.90.1`, + startup log clean (`Listening on :8080`, deadline-check scheduled for 05:00 CEST). -Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag -`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from -`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl -pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file. +## 11. Teardown + +Fill file removed; guest helper scripts and the credential file `shred`-ed; `pct fstrim 9201` returned +63.3 GiB; thin pool **30.76**; the guest is back to 1.9 G used / 64 G free. The scratch email renderer +under `hub/tmprender/` was deleted after use. The hub DB copy is scratch-only. + +## 12. Observations — NOT acted on + +1. **Only two apps are deployed on the demo box**, so "several apps" was 2 of 2 — exactly the measured + case, but a five-app demonstration would be stronger. +2. **`FormatOperatorEmail` puts an emoji in every operator subject.** Part 3 asked for none; the icon + comes from the shared formatter and changing it globally would alter every other operator mail and + its tests. The digest's own copy has no emoji. +3. **A `recorded` row is written for record-only events on the `operator` channel.** It is not an + operator *notification*, so the channel label is slightly overloaded — readable today, worth a + thought if a third channel ever appears. +4. **The customer-channel `skipped/operator_only` row no longer appears for the per-app event**, + because record-only returns before the customer leg. That is correct, but anyone grepping history + for that pattern will see it stop on 2026-08-03. diff --git a/STATUS.md b/STATUS.md index 7da1922..713aee5 100644 --- a/STATUS.md +++ b/STATUS.md @@ -14,114 +14,70 @@ A blank machine boots the Felhom disc, installs itself unattended, and is claime who sets their own password. They install apps from a catalogue of fifty-three, share files over the home network, and open apps from a launcher or a shared link. Backups run on their own to three places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can -restore files and app data from the drive alone. Proven end to end on real hardware. - -**Apps come back after a power cut.** The machine tells an app the customer switched off from one -that simply did not come back, and waits for the system to finish starting before deciding instead of -glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every -time, and an app switched off deliberately stayed off every time. +restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo +box six times, everything returned every time, and an app switched off deliberately stayed off. +Proven end to end on real hardware. ## What's broken -**The off-site copy can be erased by the machine that made it** — the credential that writes it can -also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy. -*(R-95, R-87)* - -**A full disk tells you about ONE app and silently swallows the rest.** Yesterday this was written -down the wrong way round — as *too many* emails. Measuring the receiving end reversed it: of nine -refusals the machine reported today, **two emails were sent**. When two apps are refused in the same -second you are told about one of them, and the other leaves no trace anywhere — not an email, not -even a line in the log saying it was held back. So a second app can be going unbacked-up while you -have already been told the problem is handled. It is the same fault we fixed once before for -whole-machine backups, in a second place that never opted into the fix. *(R-182)* +- **The off-site copy can be erased by the machine that made it.** The credential that writes it can + also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy. + *(R-95, R-87)* +- **Nothing else open that affects a customer.** The three faults that were on this list yesterday — + the reserve watching the wrong step, the last app whose data was never saved, and the alert that + told you about one app and swallowed the rest — are all fixed and proven on a real machine. ## What shipped recently -**Pushing the installer no longer publishes it.** The script that runs as root on a brand-new -machine was copied from the main branch and served within thirty seconds, so pushing it *was* -publishing it, with no staging and no way back but another push. It now comes from a **labelled** -version: publishing is moving the label, and undoing it is moving the label back — about half a -minute, no deploy. The website is untouched by this and still updates in thirty seconds, because a -typo fix must never need a release. Proven by actually doing it: a real push changed nothing that -anyone downloads, moving the label published it, moving it back restored the previous bytes exactly. - -**The catch that would have made it cosmetic was found and covered.** While it runs, the installer -fetches sixteen more files — not nine, and from the *agent's* repository, not the website's. They now -come from the same version of the agent the machine is installing. That closed a real fault nobody -had noticed: a new machine was getting the agent's tested program and its untested settings files, in -one install, from two different places. *(R-110, R-183)* - -**Releasing the agent now publishes it, in one command.** Putting a built agent where a new machine -can download it was a step someone had to remember, and it was forgotten three times in five days — -the last time leaving both demo machines running a version nobody could download, so a rebuild would -have quietly installed the *older* one and reported success. There is now one command that builds, -labels, publishes and then **downloads it back to check** — and a check that refuses to stay quiet if -a released version cannot actually be fetched. Proven by making CI fail on purpose and then go green -again on the same code. *(R-115)* - -**The backup partition is gone and both demo machines run on the new shape** — wiped, rebuilt and -taken through the whole customer journey on 3 August, by two deliberately different routes so the disk -shape and the delivery route are both proven. The space a backup can use went from 19 GB to 65 GB on -the small machine and 45 GB to 233 GB on the big one. Their previous demo apps and data are gone; that -was the point of a wipe, and you approved it. *(R-165, R-178)* - -**What replaced the wall now watches the right moment.** The wall was quietly keeping a runaway -backup from eating the space the machine needs to run. As first built, that replacement was checked -too late — the big write happened first, unchecked — while still promising your last good copy was -untouched. Fixed and proven on 3 August: the machine decides once, per app, **before it writes -anything**, and that one answer covers all three steps, so a refused app writes nothing, is not -restarted, and the promise is now literally true. It also stopped being blind to size. Nothing is ever -deleted to make room. *(R-181)* - -**The last of the three apps that never saved their data is fixed.** Installed nowhere, so nothing was -stranded — checked on both demo machines and in the fleet list rather than assumed. Proven by the check -that caught it, run in both directions: it clears the fixed version and still convicts the old one. -*(R-156)* - -**A filling disk warns the customer before anything breaks, and a failed backup reaches you** — the -customer while there is still room to act, naming the drive and the space left; you when one app's -backup fails, with the disk figures. The customer is deliberately not told about the second: they can -free space, but they can do nothing about a failed backup. Both proven by filling a real disk. There -are two rules and not one because the serious warning fired on free space while the disk was only 91% -full — a percentage alone would have missed it. *(R-167, R-158)* - -**The checks have two nets and the second emails you.** Every repository has one command that runs all -its checks, before every push. That one can be skipped, so the build server runs them again and emails -you on failure. It cannot *stop* a change — everything goes straight to the main copy with no review -step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)* +- **A failed backup now tells you about every app, once.** When several apps fail in one night you get + **one** email listing all of them, what kind of backup failed for each, why, and how much space is + left — instead of an email about one app and silence about the rest. Measured before the fix: nine + failures, two emails. Separately, **every failure is now written down the moment it happens**, + whether or not it is emailed, and **a message the machine decides not to send now leaves a record + saying so**. A machine choosing not to tell you and nothing happening no longer look identical. + A run that finds nothing wrong stays silent, and a backup you trigger yourself always reports. + *(R-182)* +- **Publishing became something you do, not something that happens.** The installer that runs as root + on a new machine, and the agent, both moved onto labelled versions with a way back; a check now + refuses a release nobody can download. *(R-110, R-115)* +- **The reserve now guards the step that fills the disk, and its promise is true.** It decides once + per app, before writing anything, and no longer waves an app through at 96% full. *(R-181)* +- **The backup partition is gone**; both demo machines were rebuilt on the new shape by two different + routes. Backup space went from 19 GB to 65 GB on the small machine, 45 GB to 233 GB on the big one. + *(R-165, R-178)* +- **The last of the three apps that never saved their data is fixed.** *(R-156)* +- **A filling disk warns the customer before anything breaks**, naming the drive and the space left. + *(R-167, R-158)* +- **The checks have two nets** — one before every push, one on the build server that emails you. + *(R-29, R-161, R-168)* ## What we're working on -- **Now:** nothing outstanding from today — the reserve, the last unsaved app, and both of your - decisions are all built and proven. -- **Next:** the alert that tells you about one app and swallows the second *(R-182)*. -- **After:** the off-site copy that the machine making it can still erase *(R-95, R-87)*. +- **Now:** nothing outstanding. Everything raised in the last three days is closed. +- **Next:** restore-tests scheduled around each backup rather than on a free-running timer — now + unblocked, because the off-site endpoint has the memory for it. *(R-86)* +- **After:** the off-site copy that the machine making it can still erase. *(R-95, R-87)* ## Waiting on you - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a session log; nothing suggests anyone else saw it. *(R-132)* -- **Nothing else.** Both decisions you took on 3 August are now built and proven. One small question - will come back later: the automatic check cannot see which version you have told machines to - install, only which ones exist — closing that either needs a password given to the build server or - a check inside the hub itself. Filed, not urgent. *(R-184)* +- **One small question, not urgent.** The automatic check cannot see which version you have told + machines to install, only which ones exist. Closing that needs either a password given to the build + server or a check inside the hub itself. *(R-184)* +- **One older question still standing:** whether the off-site endpoint counts as protected. It was + never named either way, and it is yours to settle. *(target-selection.md)* ## Changed since last update -- **2026-08-03** — Publishing became something you do rather than something that happens: the - installer and the agent both moved onto labelled versions with a way back, and a check now refuses - a release nobody can download. Earlier the same day: the reserve now guards the step that fills the - disk and its promise is true, and the last app whose data was never saved is fixed. All proven on - real machines, not just in tests. - Earlier the same day: both demo machines wiped and rebuilt from the new base image and taken through - set-up → install an app → back it up → restore it, with the backup space ceiling gone and measured. - -- **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in - a mode where reading a page cannot block a machine's status update; a one-word difference meant that - setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found: - the hub's own database is in no automatic backup** — it holds every machine's emergency password. - Filed, not yet fixed. - +- **2026-08-03** — A failed backup now reports every app in one email, and nothing is dropped without + a record. You rescaled the off-site endpoint to twice the memory, which closed one item and + unblocked another; the documents describing the old machine were corrected, and the audit records + of what it used to be were annotated rather than rewritten. +- **2026-08-03** — Publishing became deliberate for both the installer and the agent, with a way back + and a check that a release can actually be downloaded. +- **2026-08-03** — The reserve was fixed to guard the step that fills the disk; the last unsaved app + was fixed; both demo machines were rebuilt on the merged disk layout and proven end to end. - **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this machine and the tester's box are protected, every other box may be broken or reinstalled freely. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index c0a9f7e..c1a73da 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -86,7 +86,7 @@ | An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 | | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped | | **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait | -| **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it | +| **A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed** | controller **v0.194.0**, hub **v0.90.1** (R-158 → R-167 → **R-182**) | **PROVEN-LIVE (2026-08-03)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it. **ROW REWRITTEN 2026-08-03 (R-182) — the 2026-08-02 claim was TRUE OF ONE APP AND FALSE OF THE REST, and it is worth saying which.** The signal existed and worked; what it did not do was scale past the first failing app. Measured: nine per-app events reached the hub in one day and **two** operator mails went out, because the cooldown key is `customerID:eventType(+tier)` and this type carries `app` but no `tier` — so the first refused app took the hour and the rest were dropped **before `LogNotification`**, leaving no row on any channel. The old row said "One event per app, loop continuing", which was true of what the CONTROLLER emitted and not of what the operator received. **Now:** the per-app event is the RECORD (hub `recordOnlyEvents`: stored + logged every time, never mailed) and `backup_run_failures` is the NOTIFICATION — one mail per run listing every failed app, its leg and its reason, with the counts and free space. A suppressed operator event of ANY type now leaves a `suppressed` row naming its key. **Proven live on demo-hp 2026-08-03** by a real 64.6 GiB fill (thin pool held 30.78 → 30.78): `notification_log` shows `recovery_unit_capture_failed | operator | recorded` ×2, `backup_run_failures | operator | sent` ×1 naming BOTH apps, and `backup_run_failures | customer | skipped | operator_only`. A second run in the same hour produced a second digest; after freeing space a run completed with 2 volume dumps and **no** digest. The suppression row proved itself on an unplanned event — `disk_critical | suppressed | key=demo-hp:disk_critical` — a collapse that yesterday would have left nothing at all. **The digest's silence is safe only because** the hub's deadline check raises `expected_backup_missed` from report freshness independently of any mail (`monitor/deadline.go:396,417`); that check is load-bearing for this row | | **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.193.1** (R-165 / D-a / B2, completed by R-181) | **PROVEN-LIVE (2026-08-03) — BOTH halves** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden `; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181, CLOSED THE SAME DAY (controller v0.193.0 + v0.193.1) and the second half is now PROVEN-LIVE TOO.** The reserve became a **per-app, per-run ADMISSION decision** taken before the app's FIRST write and covering all three legs (DB dump, volume dump, capture) — they write under one per-app root, which is what lets one verdict cover them honestly — and it gained a **size term**, so an app is no longer admitted at 96% and then allowed to write 2 GB. **Re-proven by filling demo-hp deliberately, once for EACH term, using the method that found the defect.** *Headroom @ 08:59:46* (906 MB free / 99%): both apps refused, **the whole `backups/primary` tree byte-identical — `TREE_SHA` 111d1760c18d3440f700634ab325f8b8 before and after**, opengist's tar still at its original 182,272 B; **no `Stopping for safe volume dump` line at all**, which is the positive-by-absence observable that matters because that line IS present in the 08:58 baseline run; 0 volume dumps; one alert per app, HTTP 200. Space freed, re-run @ 09:01:33 → both captured normally. *Size @ 09:03:00*, reproducing the original sequence with a real 2 GiB file in opengist's volume (previous tar **2,147,666,432 B**, the exact figure the defect was measured at) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` while **privatebin was ADMITTED and dumped normally**, proving the term is per-app rather than a global halt. **The refusal's wording was NOT weakened to fit** — the behaviour moved so the wording became true, and it is verified by tree fingerprint rather than by reading the log line, which is what lied. The `fallocate` instrument was re-proven on the rebuilt box before use (5 GiB step moved guest `df` while thin-pool `data_percent` held **36.83 → 36.83**), and teardown returned the pool to **29.43%**, below its own baseline. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity | | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | | | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise | diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index dc78a02..667126a 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -574,6 +574,30 @@ because that was a different filesystem. On a merged box it can. Decision **B2** It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could only destroy a different app's only local copy. +**WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).** +The two are deliberately different mechanisms, because conflating them is how seven failures went +missing on 2026-08-03 without leaving a trace. + +| | Record | Notification | +|---|---|---| +| what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN | +| when | every time, unconditionally | at the end of a run, **only if something failed** | +| gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown | +| where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox | + +- **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that + is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's + REPORT freshness, independent of any mail the box sends. That check is load-bearing for this + design; weakening it re-opens a silent-failure path. +- **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it. + Deciding not to tell someone is itself an event worth recording. +- **Deliberate skips are not failures** and never appear in the digest — a disconnected or + decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the + operator stops reading. +- **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within + the hour, because someone pressing the button is actively trying to get a backup. The periodic + capture sweep is capped by the ordinary hourly cooldown. + **THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a **per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the diff --git a/documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md b/documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md index 642ba1a..d952d4b 100644 --- a/documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md +++ b/documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md @@ -10,7 +10,7 @@ **Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only); felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`, `internal/hub/cloudflared.go`, `internal/escrow/identity.go`). -**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`, +**Probe ends:** `felhom-hetzner` = Hetzner CX23 (**note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged**), Debian 13.4, public IPv4 `167.233.158.164`, global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** = the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven over the existing LAN SSH path; the tunnel itself always dialed **out**. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 7338c50..d26d2f1 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -26,11 +26,11 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** With the target absent the page rendered the ABSENT copy (1), the system-disk copy 0, the offer block 0 — both of E-2d's falsehoods gone. API carried `message:"A rendszermentés meghajtója nem érhető el…"` with `target:felhom-backup`. **FIXED: the third state exists.** New `BackupTargetState.TargetAbsent` separates *configured-and-gone* from *never-configured*. `Degraded` keeps its meaning (is there a problem) so the wire contract is unchanged for every consumer; `TargetAbsent` answers which problem, because the remedies are OPPOSITE — attach any second drive vs reconnect *that* one. Copy routed through `degradedMessageFor` (still one decision point) and taken **verbatim** from the hub's `backup_target_absent` email so banner and mail tell one story. **Offer suppressed on the branch itself**, deliberately not left to `firstOfferableDrive`'s `Disconnected` skip — that flag is set by R-113 in another repo, and this state must be right without it. Red-proof: deleting the branch reproduces E-2d's exact payload, offering `/mnt/felhom-drives/mentes2`, the drive that had vanished. **MinAgent unchanged 0.113.0** — R-114 reads `BackupTarget`/`MountPath`/`GuestPath`/`Role`, none of which R-113 altered, so demo-hp is not held. **NOT live-validated: Scenario C cannot occur on a healthy box.** `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC | | **R-29** | **The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it.** This is the **class**, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. `controller/scripts/docker_run_volume_path_gate.py` failed continuously from **2026-07-14 (v0.129.0)** until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" | **CLOSED — both halves shipped** (2026-08-02) | — | **This item has existed at `ROADMAP.md:158` since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding**, because it is an open item *about work not getting done* that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: **(a)** the `docker_run_volume_path_gate` finding is benign and the fix is a 3-line ALLOWLIST addition with its why — **not** a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; **(b)** the systemic half, the real item: decide where gates run (pre-push hook, `build.sh` step, or CI) and make a red gate block the train the way the Go green gate does. **Two further orphans confirmed 2026-07-29** by repo-wide grep across all file types + sibling repos + `~/.claude` settings/skills/hooks + `.git/hooks` (none non-sample) + Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) + CI-directory find (**this repo has no CI at all**) — every one of the 19 hits is a docstring, a code comment or prose, and **not one is an invocation**: `scripts/hostinstall_gates.py` — **RED today** (`hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1), the same finding as **R-94 leg (b)** — and `scripts/hub_confirm_gate.py`. Of the four gates in `scripts/`, only `site_gates.py` is mandated anywhere (`CLAUDE.md:153`) and `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance** — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either **UPDATE 2026-08-02 — leg (a) CLOSED** (`felhom-controller` `c432f70`, its own reviewed diff as specified): `appexport/estimate.go`'s `-v` is a NAMED VOLUME mounted read-only into a throwaway container, no host path, structurally identical to the allowlisted `backup/backup.go` entry — allowlisted with its why; `realVolumeSize` untouched. **Leg (b) HALF-SHIPPED:** the 'decide where gates run' ruling is now made and half-implemented — **every repo has ONE entry point** (`felhom.eu/scripts/repo_gates.py`, `felhom-controller/controller/scripts/controller_gates.py`, `felhom-agent/scripts/agent_gates.py`, `app-catalog-felhom.eu/scripts/catalog_gates.py`), each mandated in its `CLAUDE.md` and each wired to `.githooks/pre-push` via `--fast`. **THE CENSUS, which is the finding:** thirteen gate scripts across four repos; **every gate a `CLAUDE.md` names was GREEN, and two of the four nobody names were RED** — `hostinstall_gates.py` (red since 2026-07-14) and `reuse_refs_check.py` (red on all four repos); a third, `docker_run_volume_path_gate.py`, was named only in `REUSE.md:284` and was also red. Correlation with 'named in a CLAUDE.md' was exact. **STAYS OPEN for the automatic half** — a hook is per-clone and `--no-verify` skips it; the unbypassable half is CI → **R-168** **CLOSED 2026-08-02, on the demonstrated ALARM and not on a green run.** Leg (b)'s automatic half is now live: a Gitea Actions runner re-runs every repo's entry point on every push, independent of who pushed and of what they typed (→ R-168). The class this row opened — *a gate that exists, asserts something true, is red, and is invoked by nothing* — is answered at both ends: the pre-push hook refuses locally, and CI catches a `--no-verify` bypass and **emails the operator**, proven with a real red run and a provider accepted-id. What remains is not this row's finding but a working-style choice — CI reports rather than blocks because there is no merge to gate (→ R-169) | CC | | **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC | -| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC | +| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY (M) — UNBLOCKED and re-ranked 2026-08-03** | — | Trigger a tier ~24 h after **its own** newest archive. **The R-90 dependency is discharged:** that row informed the cadence because ep0 had 3.8 GB and a 14.46 GB restore read had OOMed it, so a more frequent restore-test risked knocking the offsite endpoint over. ep0 is now a **CX33 with 8 GB RAM plus a 4 GiB swapfile** (measured 2026-08-03), so headroom is no longer what sets the cadence and this can be designed on its own merits. **Do not read that as "the ceiling is gone":** the OOM was a 14.46 GB restore against 3.8 GB, and 8 GB is comfortable rather than unbounded — the restore-test cadence should still be paced, just not by fear of the endpoint | CC | | **R-87** | The restic tier is never restore-tested | **READY** | — | Design a controller-side test (no scratch-guest analogue transfers) | CC | | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC | | — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator | -| **R-90** | ep0 RAM headroom — 4 GiB swap **survived its first reboot** 2026-07-27; 3.8 GB RAM unchanged | **BLOCKED** (interim proven) | Hetzner CX33 availability — **confirmed unavailable even powered OFF**, so it is the Cost-Optimized "Limited availability", not the power state | Re-check CX33; escape hatch if urgent = CPX/CCX lines (no availability warning, higher cost) | operator | +| **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — | | **R-91** | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC | | — | First-ever **GC** on `felhom-offsite` (armed today 13:11 UTC, never run) | WATCHING | schedule | **Sun 2026-08-02 04:30 UTC** — confirm it completes | CC | | — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC | @@ -94,7 +94,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **(a) ANSWERED 2026-08-03 (P1: PASS). (b) NOT REQUIRED — operator ruling: every node is reinstalled, none migrated** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits **UPDATE 2026-08-03.** **(a) is measured and passed** — `audits/SPIKE-r165-phase0-2026-08-03.md` P1: a real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log) restore-tested on demo-hp, `pass: true`, `mount_parity: ok`, 84 s, with `mountParity` untouched. One limit stated rather than glossed: it ran with the pre-merge agent because the merged one did not exist yet, and the comparison is archive-vs-its-own-restore which never consults the host layout — **re-run it once against agent v0.120.0**, which is one command. **(b) is withdrawn, not deferred:** the operator ruled that every node is REINSTALLED rather than migrated in place (both demo boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed in a few weeks), so the in-place migration rehearsal has no consumer. Recorded explicitly rather than silently skipped | CC | | **R-183** | **A fresh install fetched the vouched agent BINARY and its sixteen CONFIG files from two different refs, and nothing compared them.** `felhom-host-install.sh` resolved the agent version from the hub manifest and sha-verified the binary — then took `felhom-agent.service`, `felhom-agent.sudoers` and fourteen more from `raw/branch/main`, i.e. whatever the agent repo's tip happened to hold at that second. One install, two refs, no comparison | **CLOSED — SHIPPED** (installer v1.23.0, 2026-08-03) | — | **Found while implementing R-110, by reading `fetch_raw`'s call sites rather than the spec's description of them** — the task said nine files from `felhom.eu`; they are **sixteen** and they come from **`felhom-agent`**. **Why it is a defect and not only untidiness:** these files are the agent's own operating surface — its systemd unit, its sudoers, its guarded wrappers — and `configs/felhom-backup-target-apply` is installed **0755 into `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`. A config newer than the binary is a root-executed artifact the vouched version was never tested against. **Not hypothetical in shape:** the fleet has shipped exactly this class before, where a config and the code that reads it moved independently. **Fixed by pinning to the agent version the install is already committed to**, on the operator's ruling: `raw/tag/v$ART_AGENT_VER/`, resolved from the hub manifest that the binary's sha is already checked against — so binary and configs now come from ONE ref by construction. **No fallback to a branch**, deliberately: a missing tag dies loudly rather than quietly serving `main`. Pinned by `hostinstall_gates.py` gate 6 (no `raw/branch/` anywhere; the `$ART_AGENT_VER` pin still present), red-proofed by reverting one of the sixteen and by removing both assertions. `felhom-agent` now carries `v` tags (`v0.120.0` created retroactively at `cd6e267`, the commit the published binary was built from; `configs/` is byte-identical there and at `main`, so nothing depended on the choice) and `release-agent.sh` creates them as part of releasing | — | | **R-184** | **Nothing prevents the hub from vouching an agent version that was never released.** The R-115 gate proves every RELEASED version is installable, but it works from git tags — so a hub artifact-manifest entry naming a version with no tag and no package is invisible to it. The installer would then die at step 5 on a virgin machine, as root | **READY (S) — NEW 2026-08-03** | — | **Filed BECAUSE the R-115 gate deliberately does not cover it, rather than leaving the gap unstated.** CI cannot check it: the hub's `/api/v1/artifacts/` answers **401** without a per-customer retrieval passphrase and Gitea's package **listing** api answers **401** without a token (both measured 2026-08-03, P-C), so a credential-free gate can ask *"is this version installable"* but never *"which version is vouched"*. **Two shapes, and the second is better:** (a) give CI a hub credential — expands what CI can reach, and is the operator's call not a gate author's; (b) **validate at vouch time, in the hub**: the operator UI's Day-0 artifact form refuses a version whose package is not downloadable. (b) fails closed at the moment of the decision, needs no new credential anywhere, and puts the check where the mistake is actually made. **Exposure is low and should be said so:** vouching is a deliberate operator action against a version they have just released, and R-115's release path now makes released-but-unpublished nearly impossible. This is the residue, not the main risk | CC | -| **R-182** | **A full disk tells the operator about ONE app and silently swallows every other app's refusal for an hour.** The hub's operator cooldown key is `customerID + ":" + eventType + cooldownTierSuffix(details)` (`hub/internal/notify/dispatcher.go:268`). `recovery_unit_capture_failed` carries **`app`** in its details and **no `tier`**, so the suffix is empty and the key contains **no app identifier**: the first refused app's alert takes the 1-hour slot and the second app's is dropped — and dropped BEFORE `LogNotification`, so it leaves **no row on any channel**. It cannot even be audited after the fact | **READY (M) — RE-SCOPED 2026-08-03, and the direction REVERSED** | — | **FILED AS THE OPPOSITE DEFECT AND THE MEASUREMENT OVERTURNED IT.** It was filed 2026-08-03 as *"the reserve re-alerts on every status refresh"* — too MANY alerts — from the controller-side observation that a second push followed 13 s after the first. That was the sending end. **Measured at the receiving end** (hub `notification_log` + `events`, read from a copy taken WITH its `-wal`, freshness confirmed by the newest row post-dating the session): **9 events received today → 2 operator e-mails sent.** **06:40:03** privatebin AND opengist both refused → **opengist e-mailed, privatebin's alert has no row at all**. **08:59:46/47** opengist AND privatebin both refused → **privatebin e-mailed, opengist's absent**. **08:59:59, 09:03:00, 09:07:06** → **no operator row whatsoever**, all inside the 1-hour cooldown opened at 08:59:47. So the controller pushing repeatedly is not the defect; the hub emitting at most one operator e-mail per customer per hour is, and the loser is silent. **This is R-97a's failure mode exactly, in a second event type.** That row's own comment states it: *"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20 for the whole hour"*. `cooldownTierSuffix` was written NARROW on purpose — empty unless the producer sends a `tier` — so no existing type's behaviour changed; `recovery_unit_capture_failed` simply never opted in. **CORRECTION OWED, and it is the reason this was worth measuring:** the 2026-08-03 R-181 report said *"one `recovery_unit_capture_failed` per app, HTTP 200"*. That was **true of what the CONTROLLER pushed** and would be read as *the operator was told about each app* — which is **false**. The distinction between an accepted event and a sent e-mail is the whole of this row. **Fix shape (NOT implemented — Part 0 was investigation only, by instruction):** let the producer opt into a per-app cooldown key, the way R-97a let the whole-guest producer opt into a per-tier one — the narrow mechanism already exists and needs no widening. **And a suppressed operator alert should leave a `skipped` row rather than nothing**, or this class stays undiagnosable from the hub's own records | CC | +| **R-182** | **A full disk tells the operator about ONE app and silently swallows every other app's refusal for an hour.** The hub's operator cooldown key is `customerID + ":" + eventType + cooldownTierSuffix(details)` (`hub/internal/notify/dispatcher.go:268`). `recovery_unit_capture_failed` carries **`app`** in its details and **no `tier`**, so the suffix is empty and the key contains **no app identifier**: the first refused app's alert takes the 1-hour slot and the second app's is dropped — and dropped BEFORE `LogNotification`, so it leaves **no row on any channel**. It cannot even be audited after the fact | **CLOSED — SHIPPED** (controller v0.194.0 + hub v0.90.0/.1, 2026-08-03) | — | **FILED AS THE OPPOSITE DEFECT AND THE MEASUREMENT OVERTURNED IT.** It was filed 2026-08-03 as *"the reserve re-alerts on every status refresh"* — too MANY alerts — from the controller-side observation that a second push followed 13 s after the first. That was the sending end. **Measured at the receiving end** (hub `notification_log` + `events`, read from a copy taken WITH its `-wal`, freshness confirmed by the newest row post-dating the session): **9 events received today → 2 operator e-mails sent.** **06:40:03** privatebin AND opengist both refused → **opengist e-mailed, privatebin's alert has no row at all**. **08:59:46/47** opengist AND privatebin both refused → **privatebin e-mailed, opengist's absent**. **08:59:59, 09:03:00, 09:07:06** → **no operator row whatsoever**, all inside the 1-hour cooldown opened at 08:59:47. So the controller pushing repeatedly is not the defect; the hub emitting at most one operator e-mail per customer per hour is, and the loser is silent. **This is R-97a's failure mode exactly, in a second event type.** That row's own comment states it: *"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20 for the whole hour"*. `cooldownTierSuffix` was written NARROW on purpose — empty unless the producer sends a `tier` — so no existing type's behaviour changed; `recovery_unit_capture_failed` simply never opted in. **CORRECTION OWED, and it is the reason this was worth measuring:** the 2026-08-03 R-181 report said *"one `recovery_unit_capture_failed` per app, HTTP 200"*. That was **true of what the CONTROLLER pushed** and would be read as *the operator was told about each app* — which is **false**. The distinction between an accepted event and a sent e-mail is the whole of this row. **Fix shape (NOT implemented — Part 0 was investigation only, by instruction):** let the producer opt into a per-app cooldown key, the way R-97a let the whole-guest producer opt into a per-tier one — the narrow mechanism already exists and needs no widening. **And a suppressed operator alert should leave a `skipped` row rather than nothing**, or this class stays undiagnosable from the hub's own records. **CLOSED 2026-08-03 — controller v0.194.0 (`88897a2`) + hub v0.90.0/.1 (`dd40f85`, `f21e7ca`), proven live on demo-hp.** **The operator ruled against the obvious fix** (putting `app` in the cooldown key): on a full disk that is one e-mail per app, the volume problem wearing the correctness problem's clothes. What shipped instead is **one digest per run** plus **a record for every failure**. **(1) The silent drop is gone.** `processOperator`'s cooldown no longer returns bare — a suppressed operator event writes a `suppressed` row carrying the message and the key that suppressed it, **for every operator event type, not only the one that exposed this**. The cooldown's duration and semantics are deliberately unchanged: this makes the drop visible, not absent. **(2) `backup_run_failures`** — one operator mail at the end of a run listing every failed app, its leg and its reason, with the counts and the free space; in `allowedEventTypes` AND `operatorOnlyEvents`, with a test demonstrating a customer who has enabled it receiving nothing. **(3) `recordOnlyEvents`** — a third routing class. `recovery_unit_capture_failed` is now stored + logged every time and never mailed: **the record and the notification are different things**, which is the durable half. A register rather than a severity downgrade, which would have relabelled a genuine failure as informational everywhere it is queried. **(4) `cooldownRunSuffix`** — a sibling of `cooldownTierSuffix`, so `tier` keeps byte-identical semantics and R-97a is untouched. A unique `run_id` per real run means the cooldown can never collapse a manual run into the nightly one (the operator ruled on that explicitly). **A GAP THE SPEC DID NOT ANTICIPATE, AND ITS FIX.** `recovery_unit_capture_failed` also fires from `GetFullStatus`'s periodic sweep, outside any run. Making it record-only would have left those failures recorded and **never notified** — a NEW silence created while closing one. So the sweep emits a digest too, deliberately with **no `run_id`**, so it stays under the ordinary hourly cooldown exactly as before while the mail now lists every failing app instead of whichever was first. **A refusal is noted ONCE, where the verdict is taken**, not at the three legs that consult it — R-181's contract is one verdict per app per run, and noting it per leg listed a single refused app three times and produced counts like *"2 of 1 apps failed"*. **Found by the digest's own test, not in review.** **LIVE PROOF, demo-hp, the same method that produced the original measurement** — a real 64.6 GiB `fallocate` to 241 MB free (thin pool held at **30.78 → 30.78**, instrument re-proven). The hub's `notification_log` for the run reads: `recovery_unit_capture_failed | operator | **recorded**` ×2 (opengist, privatebin), `backup_run_failures | operator | **sent**` ×1 listing BOTH, and `backup_run_failures | customer | **skipped** | operator_only` — i.e. two failures, one mail, both named, and the customer refused. Against yesterday's two failures → one mail naming one app and one vanishing. **Scenario D proved itself on an event I did not plan:** `disk_critical | operator | **suppressed** | operator cooldown 1h, key=demo-hp:disk_critical` — fillwatch alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now VISIBLE where yesterday it would have left nothing at all. **Scenario E:** a second run in the same hour produced a second digest. **Scenario B:** after freeing the space, a run completed with 2 volume dumps and **no digest at all**. **§3's safety property confirmed rather than assumed:** the hub's deadline check raises `expected_backup_missed`/`expected_dbdump_missed` (`monitor/deadline.go:396,417`) from report freshness and stored events, independently of any mail the box sends — so a digest's silence still means "the run finished and found nothing wrong". **v0.90.1 followed from reading the first real digest:** every app row repeated the same filesystem clause the mail already prints once, which down a list of a dozen apps pushes the part that differs off a phone screen | — | | **R-181** | **The capture floor guards the cheap leg and not the leg that fills the volume — and its refusal message asserts an invariant the code does not provide.** B2 (controller v0.192.0) is recorded on R-165 as the deliberate replacement for the bulkhead the `mp1` partition used to give. It is consulted in exactly one place — `m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which writes a manifest and a compose copy: **a few KB.** The leg that writes the bulk, `runVolumeDumps` (`backup.go:535`), has **no floor check at all** — its gates are protected-stack, volume-less, disconnected, decommissioned — and it runs FIRST, by design (*"MUST run before captureAllRecoveryUnits so the manifests enumerate the fresh tars"*, `backup.go:483`). So the write that fills the filesystem is unguarded, and the floor then refuses the write that would have cost almost nothing. **Second limb: the refusal message is false.** `recovery_unit.go:331` prints *"the previous unit is untouched and NOTHING was deleted"*. Nothing was deleted — true. Untouched — **measured false**: privatebin's `volume-dumps/privatebin_privatebin_data.tar` went `26c546c2…` → `b538ab89…` and opengist's went **182,272 B → 2,147,666,432 B**, both rewritten by the earlier leg, while each unit's `manifest.json` kept `created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so a unit's payload can be swapped under a stale descriptor and **nothing in the unit can detect it** | **CLOSED — SHIPPED** (controller v0.193.0 + v0.193.1, 2026-08-03) | unblocks **R-165** | **FIRST LIVE FIRING OF B2, and it is why the runbook asked for one.** Proven on demo-hp 2026-08-03 06:40:03 on a box reinstalled from the merged golden (R-178). Method: a real 2 GiB file in opengist's data volume, then `fallocate` to bring the filesystem to 96 % used / 3.0 GiB free — both floor terms deliberately still clear, so the run started. **The `fallocate` instrument was proven before use** (5 GiB moved guest `df` 977M→6.0G while thin-pool `data_percent` stayed 29.03 → 29.03: zero blocks allocated), because demo-hp's thin pool is 53.93 GiB and a real fill to 97 % of a 70 G volume would have exhausted it and corrupted every guest on the box including the `drill-r50` fixture. Sequence observed: opengist's volume dump wrote **2.0 GB unguarded** → free fell to 1.0 GB → **both** apps' recovery-unit captures were then REFUSED on the `1.0 GiB free` term, each pushing `recovery_unit_capture_failed` (severity `error`) to the hub, accepted HTTP 200. **What DOES hold: it refuses per app rather than aborting the run, it never deletes, and the alert reaches the operator.** **Fix shape, not written this session by design (§7 of the runbook):** the floor belongs before the write in `runVolumeDumps` too, the message must stop claiming what the earlier leg has already falsified, and per `CLAUDE.md` *"a comment asserting an invariant needs a test pinning it"* the pinning test must assert the **consequence** (after a refusal, is the previous unit's payload byte-identical?) and not the mechanism. **Class:** the sixth entry in `CLAUDE.md`'s own table of shipped guarantees the code did not provide — found, as four of those were, only on live hardware. **CLOSED 2026-08-03 — controller v0.193.0 (`fef07c3`) + v0.193.1 (`6c43bf6`), proven live on demo-hp.** **The fix is ONE admission verdict per app per run** (`internal/backup/admission.go`), taken before that app's FIRST write and consulted by all three legs — the three write under one per-app root (`appbackup.RecoveryUnitPath`), which is exactly why one verdict can honestly cover them. **Decided LAZILY at the app's first write, never once at run start**: app A's dump can put app B under the reserve, so a run-start verdict would read a disk that no longer exists — the same class of mistake one level up. **Never re-decided between an app's own legs** (that IS the split this closes) and **reset per run**. Placed ahead of `DumpAppVolumesSafe`, which stops the stack as its first act, so a refused app is never bounced; placed AFTER the volume-less check, which has no write to gate. Exactly ONE operator alert per refused app per run. Leg order unchanged. **The floor is now SIZE-AWARE**, which is the second half of the defect: it asks whether THIS app's write would cross the reserve, not only whether the filesystem is already below it — the term whose absence admitted an app at 96% and then let it write 2 GB. Estimate = the app's previous `.sql`+`.tar` on disk; **no history → headroom-only** deliberately, or the first backup becomes the one that can never happen, and the alert says so. **A container-based `du` was MEASURED and rejected, not assumed**: 66 timed runs on demo-hp guest 9201, **median ~355 ms/volume (341–404)** on volumes holding tens of KB — the cost is container start-up, not the walk. Decisive on top: `docker run` needs the writable layer, so the instrument can fail under exactly the pressure the reserve exists to handle; and the previous-dump estimate measures the ARTIFACT that will be written rather than the live volume. **THE MESSAGE WAS NOT WEAKENED — the behaviour moved so the wording became true**, and it is checked by sha256 tree fingerprint, not by reading the log line (which is what lied). **LIVE PROOF, demo-hp guest 9201, the same method that found it.** The `fallocate` instrument was RE-PROVEN on the rebuilt box before use (guest `df` 1.2G→6.2G on a 5 GiB step while thin-pool `data_percent` stayed **36.83 → 36.83**: zero blocks allocated), because a real fill of a 70 G volume would exhaust the 53.93 GiB pool. **Headroom term @ 08:59:46** — 906 MB free / 99%: both apps refused, **`TREE_SHA` 111d1760c18d3440f700634ab325f8b8 IDENTICAL before and after** (10 files, incl. opengist's tar still at 182,272 B — R-181's own 'before' figure), **no `Stopping for safe volume dump` line at all** (it is present in the 08:58 baseline, which is what makes its absence evidence), 0 volume dumps, one `recovery_unit_capture_failed` per app HTTP 200. **Freed and re-run @ 09:01:33** — both captured normally. **SIZE term proven separately @ 09:03:00**, reproducing the original sequence with a real 2 GiB file in opengist's volume (its previous tar then **2,147,666,432 B**, the exact live figure) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` — *"this app's last backup was 2.0 GB and writing it again would cross the reserve"* — while **privatebin was ADMITTED and dumped normally**, proving the term is per-app and not a global halt. **Teardown complete**: fill removed, planted file removed, `pct fstrim 9201` returned 67.5 GiB, pool **29.43%** (below the 36.83% baseline), tree byte-identical to the pre-test fingerprint. **v0.193.1 shipped in the same session**, found by this very proof run: the estimate was rendered fixed to 2-decimal GiB, so opengist's real **178 KB** printed as `estimated 0.00 GiB write` — which reads as *no estimate was available* and is the opposite of what happened. Rendering moved to `humanizeBytes`; arithmetic unchanged. Re-verified live: `estimated 178.0 KB write`. **11 new tests + 4 red-proofs**, each demonstrated failing then restored: both dump-leg gates removed (= v0.192.0) → Scenario A red with the tree shown changing; the size term removed → Scenario D red; a prune injected into the refusal path → Scenario F red; the floor moved above the warning band → Scenario G red. **Recorded honestly: the specified Scenario-F mutation (remove the reserve entirely) did NOT turn F red** — removing it makes every app write, which overwrites and adds but deletes nothing, so a deletion-watching test correctly stays green; the prune mutation is the one that proves the assertion. The DB leg cannot run without Docker, so its gate is pinned by an **AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne` — `strings.Contains` is insufficient, a commented-out call still contains the string. **§3's correction CONFIRMED in passing and not chased**: `restore_points.go:57-59` takes the manifest mtime then `newestArtifact` over `.sql` and `.tar`, so the newest of the three wins — the restore point does NOT show a stale timestamp. **New finding from the live run → R-182.** | — | | **R-180** | **`--archive-storage` is accepted without checking the agent's token will ever be granted on it, and the failure lands at step 8/8 — after the root@pam password has already been rotated.** `felhom-host-install.sh` validates the archive storage EXISTS (`pvesm status --storage`, `:1583`) and that the golden volid RESOLVES on it (`:1661`), both in pre-flight. It never checks that storage against the ACL set it is about to grant, which is the fixed default `local local-lvm felhom-pbs` (`--acl-storages`, which `runbooks/day0-install.md` tells the operator **not** to pass). A storage outside that set therefore passes every pre-flight gate and dies at the last step | **READY (S) — NEW 2026-08-03** | — | **Hit live on demo-hp 2026-08-03** during R-178 Phase A, self-inflicted and therefore a clean demonstration: the golden was staged on `felhom-backup` (the enrolled NVMe, where the box's vzdumps live) and `--archive-storage felhom-backup` passed. Pre-flight passed; steps 1–7 ran; step 8 returned `reconcile: bring-up restore: proxmox: POST /nodes/felhom-host/lxc -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)`. **The cost is the ORDER, not the error** — by the time it fires, step 2 has minted the PVE token, step 4b has **rotated root@pam and vaulted it** (so the old console password is already dead), and step 5 has installed the agent. Recovery was `--resume` after moving the golden to `local`, which worked cleanly. **This is statically checkable in pre-flight**: `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a one-line assertion over two variables both known at `:1583`. Same class as R-29 — the checkable thing that nothing checks | CC | | **R-179** | **`--uninstall` leaves the NAS network-storage systemd units behind, with the automount in `failed` state and the parent bind still mounted.** The teardown's residue-diff provenance (`day0-install.md` Part E: *"a full-filesystem diff against the pre-install baseline showed zero Felhom-named leftovers"*) is from **v1.9.1**, which predates the NAS network-storage feature. A box that has ever had a network share configured keeps `/etc/systemd/system/mnt-felhom\x2ddrives-.mount` and `.automount` after a full uninstall | **READY (S) — NEW 2026-08-03** | — | **Observed on demo-hp 2026-08-03** after `--uninstall --vmid 9201`: `mnt-felhom\x2ddrives-Felhom\x2dShare.automount` **loaded failed failed**, its `.mount` `loaded inactive dead`, and `mnt-felhom\x2ddrives.mount` still `active mounted` — the uninstall's own output had warned `/mnt/felhom-drives/Felhom-Share is busy — NOT forcing` and `/mnt/felhom-drives root bind left mounted`, which is correct behaviour (it never forces an unmount) but is not teardown. Cleared by hand before the reinstall: stop both units, remove both unit files, `daemon-reload`, unmount the autofs then the parent. **NEGATIVE CONTROL, same day:** demo-felhom's uninstall left **nothing** (`ls /etc/systemd/system | grep -i felhom` → only the unrelated `felhom-bootstrap.service`; no felhom mounts) — because that box had no network share configured. **So the residue is conditional on the feature having been used, which is exactly why a diff taken on a box that never used it reported clean.** `felhom-bootstrap.service` is NOT residue — it is the ISO first-boot unit, `disabled`+`inactive`, exactly-once and already fired | CC | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 799c979..23e145e 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -144,7 +144,7 @@ | R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. | | R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. | | R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). | -| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. | +| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — UNBLOCKED 2026-08-03 (R-90 closed: ep0 is now a CX33, 8 GB + 4 GiB swap, so endpoint headroom no longer sets the cadence — though 8 GB is comfortable, not unbounded: the original OOM was a 14.46 GB restore)** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. | | R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. | | R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | **Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change)** — **Part 1** added the failure breaker: consecutive failures tracked per TARGET, backoff `15m→30m→1h→2h→4h` capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to `TriggerNow`, and `stillRunning` is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. **Part 2 is agent-side and needs a wire change** — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely. @@ -175,7 +175,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the | R-162 | **`docker diff` is the gate's only witness and its failure mode is quiet** | XS | WATCHING — 2026-08-02 | A limitation, not a defect. The gate's power is `docker diff` excluding mounted paths; on a driver where it is unsupported or lies, the gate degrades to mount-occupancy + writability **and would not say so**. It fails closed (the canary self-test stops reporting BROKEN and the gate then refuses to report), but the message blames the prober rather than the driver. Revisit only if a non-overlay driver ships | | R-163 | **`mp1` is RETENTION, not staging — and it is sized as if it were neither** | M | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | Settled by reading: a unit is the KEPT copy on the app's **own** drive (`backup.go:245-255`), falling back to the system SSD for driveless apps — *"the SSD-only system-data fallback"* (`appbackup/paths.go:26-27`); **no post-copy deletion**, the only prune is F5 residue-on-old-drives (`backup.go:1053-1112`). So `mp1` **20 G** retains every driveless app's unit while `mp0` permits **50 G** of volumes, and a DB app's unit is up to **~2×** its data (measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to 0 (`felhom-agent/.../main.go:178`), not derived from the drive; demo-hp's real guest ships 50G/20G. **`mp1` gates the whole chain** — Tier-2 mirrors the unit *"(always)"* (`tier2.go:302,368`) and Tier-3 carries it. Bounded: a unit is volume tars + DB dumps only, **never `mp8` userdata** (`recovery_unit.go:20-25`). **No number proposed.** → `07-backup-architecture.md` §7.5 | | R-164 | **C2's chain — the DB volume tar cannot be dropped until a sound dump predicate exists** | S | BLOCKED — on the predicate (2026-08-02) | The unit holds a volume tar **and** a SQL dump and the restore uses both: the dump is authoritative and replayed after the tar so it WINS (F17), with only the DB service up (R-47) — `restore_unit.go:262-266`. Dropping the DB tar would halve DB-app units and close R-127(b)'s initdb-skip trap. **The obvious gate is dead, measured:** `ValidateDump`'s empty-`accounts` warning was **correct** (the DB truly had 0 rows; seeding one stopped the warning and put the row in the dump) — but **a fresh appliance legitimately has zero accounts**, so gating on it blocks every new customer's first backup. Order: sound predicate (dump vs **live** per-table counts) → warn→gate → tar-drop. Pairs with **R-127** | -| R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console | +| R-90 | ~~**ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86**~~ | S | **CLOSED 2026-08-03 — the operator rescaled ep0 to a CX33.** MEASURED on the box, not read from an invoice: `Mem: 7757` MB (**8 GB**, was 3.8), `nproc` **4**, and the 4 GiB swapfile added 2026-07-27 **survived the resize** and is active (checked because a resize is a stop/start, so "still there" was an assumption until measured). **The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only — so no disk figure in any runbook needed correcting. Unblocks R-86 | Runbooks corrected; SPIKE/evidence documents ANNOTATED not revised. `target-selection.md`'s "D-d did not name ep0 either way" stays open — it is the operator's | | R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` | | R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect | | R-93 | **`drill-r50` is both a blocked customer and the only drift fixture** | XS | idea — 2026-07-27 | The drill customer is blocked in the hub (so it stops alarming) yet it is also the only record exercising the endpoint-drift path R-77 added. Blocking hides it from `GetActiveCustomerIDs`, so the fixture it provides is silently inert — a monitor with no live subject reads exactly like a monitor that passes. Decide: retire it and build a synthetic fixture, or unblock it and silence per-customer instead (the operator has a per-alert silencing feature planned). Related to the R-50 drill VM, now shut down | diff --git a/documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md b/documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md index 16759c4..b91bbbe 100644 --- a/documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md +++ b/documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md @@ -3,7 +3,8 @@ **Class:** supervised operational run. **No repo version bump** — the only commits are this record and the capacity note. **Nothing was deleted.** -**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg. +**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner **CX33 (4 vCPU / 8 GB RAM)**, Nuremberg. +> **Rescaled 2026-08-03** from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. **The 40 GB local disk did NOT change** — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize. **Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom` → **`/mnt/pbs-datastore`** (name unchanged). **Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC. diff --git a/documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md b/documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md index e3f2056..d739899 100644 --- a/documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md +++ b/documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md @@ -229,7 +229,9 @@ as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as `keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few minutes of ep0 CPU per weekly backup. -> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup +> Watch item (**superseded 2026-08-03**: ep0 is now a **CX33, 8 GB RAM**, and it HAS a 4 GiB swapfile +> which survived the resize — so the pressure below is much reduced, though the shape of the concern +> stands). As written: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup > window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see > `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box. @@ -307,7 +309,7 @@ Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13 watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here. 4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to it. Not a fault, but it cannot be used as write-proof evidence for small deltas. -5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM. +5. ~~**ep0 has no swap** (3.7 GB CX23)~~ — **corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile.** See the volume runbook's OOM for the original incident. ## 11. Observations diff --git a/documentation/runbooks/offsite-endpoint.md b/documentation/runbooks/offsite-endpoint.md index 486c1eb..6ea7aa0 100644 --- a/documentation/runbooks/offsite-endpoint.md +++ b/documentation/runbooks/offsite-endpoint.md @@ -5,7 +5,8 @@ > firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM > re-creates the endpoint from nothing (that is the DR story, step 8). > -> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23, Debian 13, +> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled +> to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13, > `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a > later re-run of this runbook on a production VM. > @@ -31,7 +32,7 @@ Parameters used throughout (adjust for a new endpoint): points at nothing (live-run finding). Home-resolver propagation can lag public DNS by minutes — a client-side `wg-quick up` that fails to resolve right after record creation just needs a retry. -- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway CX23), not +- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not any production box. ## 1. Base (on the box, as root) diff --git a/documentation/runbooks/target-selection.md b/documentation/runbooks/target-selection.md index 20672ec..990b96d 100644 --- a/documentation/runbooks/target-selection.md +++ b/documentation/runbooks/target-selection.md @@ -98,7 +98,8 @@ enrolled host. No access route from DooPlex, and nothing here needs one. ### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either** Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB -path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune +path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a +4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy — customer documents and photos, on a credential that can still delete (R-95). **Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that diff --git a/documentation/tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md b/documentation/tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md index 83e3653..b24fc30 100644 --- a/documentation/tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md +++ b/documentation/tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md @@ -139,7 +139,7 @@ is not reachable, for two reasons that are each **already-recorded deliberate po `endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`). So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is -Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off* → +Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90 — **note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off* → no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits. Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the