docs: R-86 closed and proven live; ep0 recorded as protected; R-185/186/187 filed
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal reading is never true on a daily tier); R-87 re-ranked UP because R-86 built most of what it waited for; R-185 (the agent cannot list demo-felhom's host backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's sha is not reproducible from its tag), R-187 (R-115's publish leg had never actually run) filed. R-184 was the highest ID in use. - ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row itself proposed — which would have been the never-fires version. - 07-backup-architecture: new contract section — restore-testing is per ARCHIVE GENERATION, with the trap and what did not change (S-1). - 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on the 635 s due-triggered offsite run, with the restart and teardown evidence. - CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines). Numbered 17/18 because S-14 and S-15 were already duplicated in the file. - STATUS: rewritten for the operator, trimmed back to one screen.
This commit is contained in:
+39
@@ -17,6 +17,45 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-17 — restore-testing is PER ARCHIVE GENERATION, and the hub's staleness window follows each
|
||||
tier's own rhythm (2026-08-03, R-86; agent v0.121.0 + hub v0.91.0).**
|
||||
|
||||
*The rule.* Let **A** be the newest archive on a tier that has settled for at least the settle lag
|
||||
(24 h). The tier is **DUE** when A exists and **A has not already been proven**. The daemon-start
|
||||
ticker survives only as the **evaluation interval** (6 h). A daily tier is proved daily on yesterday's
|
||||
archive; a weekly tier weekly on its own; a tier with no archive is UNKNOWN, never a fault.
|
||||
|
||||
*The trap, written down so it is not reintroduced.* The literal reading of R-86 — *"due when the
|
||||
newest archive is ≥24 h old"* — is **never true on a daily tier**, because a new archive resets the
|
||||
newest-archive age to zero long before it reaches the lag. It would have switched restore-testing off
|
||||
for the tier that matters most, silently. Red-proved at 0 runs over 5 simulated days
|
||||
(`felhom-agent/internal/backup/restoretest_due_test.go`).
|
||||
|
||||
*What the state holds now.* `RestoreTestState` records **which archive** was proven, not just when a
|
||||
tier passed — a timestamp cannot answer *"have we proven THIS archive"*. A pre-R-86 file keeps its
|
||||
time (rotation ordering survives a deploy) and yields no proven archive, so each tier is due exactly
|
||||
once after the upgrade.
|
||||
|
||||
*The old config key.* `backup.restore_test_cadence_seconds` is DEPRECATED. **Negative still disables**
|
||||
verbatim; a positive value now seeds the **settle lag** only, and the daemon WARNs once at start-up
|
||||
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
|
||||
(default 24 h). It is deliberately NOT carried into the evaluation interval.
|
||||
|
||||
*The hub half is not optional.* `restoreProvenStaleAfter` was a flat 7 days **derived from the cadence
|
||||
R-86 removes**, and a healthy weekly tier's proof age reaches EXACTLY 168 h just before its next
|
||||
proof — it sat ON the line. `restoreProvenWindow(tier, observed, ok)` now takes the tier's own
|
||||
observed archive interval × 4 generations, floored at 7 days, capped at 12 days (strictly inside the
|
||||
2-week offsite retention), falling back to the tier's **declared** rhythm (`backupStaleAfter` 26 h /
|
||||
`offsiteBackupStaleAfter` 8 d — the backup-freshness checker's own thresholds) when history is too
|
||||
short to observe one. Shipping Part 1 alone would have produced a nightly false alarm.
|
||||
|
||||
**S-18 — `ep0` is Tier 2, PROTECTED (operator ruling, 2026-08-03).** D-d named two protected machines
|
||||
and did not name ep0 either way; `runbooks/target-selection.md` carried the question in writing for
|
||||
two days. The ruling **extends D-d's protected list to three machines**: DooPlex, Peti's cluster,
|
||||
**ep0**. It is a classification, not a new set of prohibitions — destroying datastores, prune jobs,
|
||||
tunnel config or nftables rules was already forbidden by what it would destroy, and the ordinary
|
||||
off-site READ a restore-test performs remains permitted.
|
||||
|
||||
**S-13 — the `mp1` merge landed, and the variant was chosen on measurement (2026-08-03, R-165 / D-a).**
|
||||
The appliance's two data volumes are one. **Variant V-c**: the volume mounts at the NEUTRAL path
|
||||
`/var/lib/felhom`, and both `/var/lib/docker` and `/mnt/sys_drive` are binds of subdirectories of it.
|
||||
|
||||
@@ -1,188 +1,130 @@
|
||||
# REPORT — R-182: one email per backup run, nothing dropped without a trace + ep0 rescaled (R-90, R-86)
|
||||
# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)
|
||||
|
||||
**Date:** 2026-08-03 · **Repos:** `felhom-controller` **v0.194.0** (`88897a2`), `felhom.eu` hub
|
||||
**v0.90.0 → v0.90.1** (`dd40f85`, `f21e7ca`)
|
||||
**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1**
|
||||
(`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not
|
||||
after it. Ruling recorded: **ep0 is Tier 2, protected**.
|
||||
|
||||
## 1. Baselines — re-read on arrival
|
||||
## 1. Baselines, re-read on arrival
|
||||
|
||||
| Repo | @ arrival | Version | Matched §1? |
|
||||
| Repo | `main` @ commit | Version | Matched §1? |
|
||||
|---|---|---|---|
|
||||
| `felhom-controller` | `db0d4b129d3b` | v0.193.1 | **yes** |
|
||||
| `felhom.eu` | `7dc1744eece3` | hub v0.89.0 | **yes** — my local clone was one commit behind origin (another session had deleted 24 stale `REPORT-*.md` files); pulled before starting |
|
||||
| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** |
|
||||
|
||||
## 2. §3's safety property — confirmed at file:line
|
||||
The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in
|
||||
`REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and
|
||||
`REPORT.md` is overwritten every session, so the per-repo history under-reported what was running.
|
||||
|
||||
`hub/internal/monitor/deadline.go:396` raises `expected_backup_missed` and `:417` raises
|
||||
`expected_dbdump_missed`, both from `assessBackupFreshness(reportJSON,…)` and `GetEventsByType` — the
|
||||
box's **report freshness and stored events**, never from an email. So the hub notices a run that never
|
||||
finished **independently of any mail the controller chooses to send**, and a digest's silence still
|
||||
means "the run finished and found nothing wrong". **The whole design rests on this; weakening that
|
||||
check re-opens a silent-failure path.**
|
||||
## 2. Why Part 2 could not ship later
|
||||
|
||||
## 3. Files
|
||||
The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved
|
||||
weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment
|
||||
derived that number *from the cadence R-86 removes*.
|
||||
|
||||
**Hub:** `internal/notify/dispatcher.go` (suppression row, `cooldownRunSuffix`, `recordOnlyEvents`,
|
||||
`operatorOnlyEvents`) · `internal/notify/templates.go` (digest renderer, `trimRepeatedUsage`) ·
|
||||
`internal/api/handler.go` (allowlist) · `internal/store/store.go` (status doc) · two new test files ·
|
||||
`recovery_unit_dispatch_test.go` (updated, deliberately) · `hub/CHANGELOG.md` · `manifests/hub.yaml`
|
||||
The brief said a weekly tier would "sit exactly on that line". That is literally true, and the
|
||||
arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the
|
||||
current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false
|
||||
by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late,
|
||||
one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
|
||||
|
||||
**Docs:** `CONTEXT.md` (S-16) · `STATUS.md` · `documentation/architecture/07-backup-architecture.md`
|
||||
(the alerting contract, S-1) · `00-capability-map.md` · `backlog/{OPEN-ITEMS,ROADMAP}.md` · five
|
||||
runbooks corrected · two audit/evidence documents annotated
|
||||
|
||||
## 4. Tests and red-proofs
|
||||
|
||||
`go build && go vet && go test ./...` in `hub/` → **17 packages ok, rc=0**. `repo_gates.py --fast` →
|
||||
all 5 gates OK.
|
||||
|
||||
| # | Mutation | Result |
|
||||
|---|---|---|
|
||||
| D | the suppression row removed (back to the bare `return`) | **RED** — `want 1 'suppressed' operator row, got 0` |
|
||||
| E | `cooldownRunSuffix` removed from the key | **RED** — `operator mails = 1, want 2`; the manual run's digest is swallowed |
|
||||
| G | the digest removed from `operatorOnlyEvents` | **RED** — and it showed **a customer actually receiving the operator digest**, demonstrated rather than argued |
|
||||
| A | (controller) all four `noteFailure` feeds removed | **RED** |
|
||||
| I | (controller) the `main.go` seam commented out | **DID NOT FAIL FIRST TIME** — see below |
|
||||
|
||||
**Red-proof I is the one worth reading.** The AST test walked the backup package but not `main.go`, so
|
||||
the seam could be disconnected with the whole suite green. Per §9.11 that is not a pass: **the test was
|
||||
fixed and the mutation re-run**, at which point it failed correctly. The string `SetRunSummaryNotify`
|
||||
is still present in the commented-out file, which is exactly why this is an AST walk.
|
||||
|
||||
**One passing test was changed, deliberately.**
|
||||
`TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer` required the OPERATOR to be emailed a per-app
|
||||
capture failure — correct when that event was the only signal, wrong now that it is the record and the
|
||||
digest is the notification. Its **customer-safety claim is unchanged** and is why the test still
|
||||
exists; the operator assertion is inverted with the reasoning written in place, and it now asserts the
|
||||
`recorded` row instead. R-158's guarantee **moved rather than weakened**.
|
||||
|
||||
## 5. The live run
|
||||
|
||||
Demo-hp guest 9201, filled for real to **241 MB free / 100% used** (thin pool held **30.78 → 30.78**).
|
||||
Triggered through `POST /api/debug/backup/dbdump` — the exact endpoint the debug UI button calls.
|
||||
|
||||
**The hub's `notification_log`, quoted:**
|
||||
## 3. The window as derived
|
||||
|
||||
```
|
||||
11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "opengist" …
|
||||
11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "privatebin" …
|
||||
11:50:45 | backup_run_failures | operator | sent | | 2 of 2 apps failed to back up in this manual run: opengist, …
|
||||
11:50:45 | backup_run_failures | customer | skipped | operator_only | 2 of 2 apps failed …
|
||||
11:51:00 | disk_critical | operator | suppressed| operator cooldown 1h, key=demo-hp:disk_critical | …
|
||||
11:51:00 | disk_critical | operator | sent | | …
|
||||
interval = declaredArchiveInterval(tier) // host 26h, offsite 8d
|
||||
if observedOK && observed > interval { interval = observed } // observation may only WIDEN
|
||||
window = clamp(4 * interval, floor 7d, cap 12d)
|
||||
```
|
||||
|
||||
Two failures → **both recorded**, **one** email naming **both**, customer refused with the reason
|
||||
logged. Against the measured previous behaviour: two failures → one email naming one app, one leaving
|
||||
no trace.
|
||||
- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same
|
||||
tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience.
|
||||
- **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very
|
||||
thresholds the backup-freshness checker already judges those tiers against. Not a second opinion:
|
||||
if the two checkers disagreed about how often a tier receives an archive, one would be alarming on
|
||||
the other's model.
|
||||
- **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful
|
||||
`backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both*
|
||||
arrays, and classifying by array membership would attribute an offsite archive to the host tier).
|
||||
Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
|
||||
- **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before.
|
||||
- **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is
|
||||
never called stale against an archive PBS has already pruned.
|
||||
|
||||
**Scenario D proved itself on an event I had not planned.** `disk_critical` alarmed on two
|
||||
filesystems; the second was collapsed by the cooldown, and that collapse is now **visible with its
|
||||
key**. Yesterday it would have left nothing at all — which is the entire finding, demonstrated by
|
||||
accident on a different event type.
|
||||
| tier | interval used | window | proof age in health | verdict |
|
||||
|---|---|---|---|---|
|
||||
| daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before |
|
||||
| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin |
|
||||
| newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes |
|
||||
|
||||
**Scenario E:** a second run in the same hour (11:51:28) produced a second digest.
|
||||
**Scenario B:** after freeing the space, the run completed with 2 volume dumps and **no** digest line.
|
||||
## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box
|
||||
|
||||
## 6. The email as shipped
|
||||
demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and
|
||||
`2026-07-28T04:49:43Z` — **8 h 54 m apart**, because one is a healing artefact and the other a real
|
||||
weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor
|
||||
lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm
|
||||
this whole task exists to prevent would have returned within a week, on the box it had just shipped
|
||||
to.**
|
||||
|
||||
```
|
||||
SUBJECT: [Felhom] 🔴 demo-hp: 2 of 2 apps failed to back up (manual run)
|
||||
Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means
|
||||
nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real
|
||||
information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared
|
||||
rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a
|
||||
signal meaning *unverified* — *broken now* is `restore_test_failed`, immediate and untouched.
|
||||
|
||||
Customer: demo-hp
|
||||
Event: backup_run_failures
|
||||
Severity: error
|
||||
Time: 2026-08-03 13:52 CEST
|
||||
Message: 2 of 2 apps failed to back up in this manual run: opengist, privatebin
|
||||
## 5. Kept, because it was earned
|
||||
|
||||
FAILED: 2 of 2 apps attempted in this manual run.
|
||||
- **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged).
|
||||
- **The stale signal stays edge-triggered** (`staleStates` untouched).
|
||||
- **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken,
|
||||
the other that it is unverified, and the second is the one that quietly becomes the first.
|
||||
- **Every reason string now states the window it was judged against** — R-100's corollary: when a
|
||||
verdict changes what it counts from, the alarm text must change with it, or an operator reads
|
||||
"limit 168h" under a tier judged at 288 h and dismisses a true alarm.
|
||||
- **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window
|
||||
and to see two generations of a weekly tier.
|
||||
|
||||
opengist whole app (refused before any write) refused: backing up this app would leave the
|
||||
filesystem below the reserve (reserve: 97% used
|
||||
or 1.0 GiB free; the filesystem is already below
|
||||
it, before this app's estimated 178.0 KB write)
|
||||
privatebin whole app (refused before any write) refused: … estimated 2.5 KB write
|
||||
## 6. Tests and red-proofs
|
||||
|
||||
Filesystem: /mnt/sys_drive — 65.0/68.7 GB used (95%), 0.2 GB free
|
||||
| Test | Asserts | Mutation | Observed |
|
||||
|---|---|---|---|
|
||||
| `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL** — `week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` |
|
||||
| `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval` → `observed > 0` | **FAIL** — `window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` |
|
||||
| `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass |
|
||||
| `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass |
|
||||
| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged |
|
||||
|
||||
Every failure above is also recorded individually in the notification log,
|
||||
whether or not this mail was sent.
|
||||
**A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular
|
||||
weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on
|
||||
exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while
|
||||
looking like proof. The jitter is what makes it a test — and it is also the truth about the old
|
||||
constant.
|
||||
|
||||
Dashboard: https://hub.felhom.eu/customers/demo-hp
|
||||
```
|
||||
## 7. Deployment
|
||||
|
||||
**v0.90.1 came from reading exactly this.** In the first rendering every app row also ended with
|
||||
`— /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free` — the same clause the `Filesystem:` line
|
||||
already carries. On two apps that is untidy; down a list of a dozen it is forty characters twelve
|
||||
times, pushing the part that *differs* off a phone screen at 07:00. The reserve's message is unchanged
|
||||
(it is right for a single-app alert); the digest trims the duplicate when rendering, and only an exact
|
||||
`— <target path>:` suffix.
|
||||
|
||||
## 7. §8.1's table, in practice
|
||||
|
||||
The run refused **both** deployed apps on the reserve → both in the digest. **No** deliberate skips
|
||||
occurred in the live run (no drive was disconnected or decommissioned on that box), so that row of the
|
||||
table was exercised by test only — `TestRunSummary_DeliberateSkipsAreNotFailures`, which marks one app
|
||||
disconnected and one decommissioned through the real `settings` store and asserts **no digest at all**.
|
||||
|
||||
## 8. ep0 — measured, then corrected
|
||||
|
||||
| | measured on the box |
|
||||
| Step | Evidence |
|
||||
|---|---|
|
||||
| hostname | `felhom-hetzner` |
|
||||
| RAM | **7757 MB (8 GB)** — was 3.8 |
|
||||
| vCPU | **4** |
|
||||
| swapfile | **survived the resize** — `/swapfile`, 4 GiB, `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`) |
|
||||
| local disk | `/` 38 G, 58% used — **UNCHANGED**, as expected for a CPU/RAM resize |
|
||||
| datastore volume | `/mnt/pbs-datastore` 98 G, 8% — unaffected |
|
||||
| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` |
|
||||
| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` |
|
||||
| ArgoCD deliberate sync | `sync=Synced health=Healthy` |
|
||||
| Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` |
|
||||
| Startup log | checkers initialized; `Listening on :8080` |
|
||||
|
||||
The swapfile was **checked rather than assumed**: a resize is a stop/start, so "the interim lever is
|
||||
still in place" was an assumption until measured.
|
||||
## 8. Part 3 — the ep0 ruling, recorded
|
||||
|
||||
**Corrected** (five places, three of which the spec's list did not name — found by searching):
|
||||
`RUNBOOK-ep0-datastore-volume-2026-07-27.md:6` · `RUNBOOK-pbs-prune-serverside-2026-07-27.md:232,:310`
|
||||
· `runbooks/offsite-endpoint.md:8,:34` · **`runbooks/target-selection.md:101`**.
|
||||
`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"*
|
||||
for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table
|
||||
names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**.
|
||||
`CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex,
|
||||
Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores,
|
||||
prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the
|
||||
ordinary off-site read a restore-test performs remains permitted** — which this session then exercised.
|
||||
|
||||
**Annotated, body untouched** (they record what was true when written, and that is their value):
|
||||
`audits/SPIKE-connectivity-wireguard-2026-07-03.md:13` ·
|
||||
`tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md:142`.
|
||||
## 9. Registers and docs
|
||||
|
||||
**No disk figure was touched anywhere** — the 40 GB local disk did not change.
|
||||
|
||||
## 9. Register
|
||||
|
||||
| ID | Outcome |
|
||||
|---|---|
|
||||
| **R-182** | **CLOSED — SHIPPED**, with the live evidence above |
|
||||
| **R-90** | **CLOSED** — ep0 rescaled to a CX33; measured on the box, swapfile confirmed surviving |
|
||||
| **R-86** | **READY, unblocked and re-ranked** — endpoint headroom no longer sets the restore-test cadence. Stated honestly: 8 GB is comfortable, not unbounded (the original OOM was a 14.46 GB restore), so the cadence should still be paced — just not by fear of the endpoint |
|
||||
|
||||
**No new IDs were minted**, so no grep was needed. R-184 remains the highest.
|
||||
|
||||
**`target-selection.md`'s ep0 protection question is STILL OPEN and deliberately untouched** —
|
||||
*"D-d did not name ep0 either way. Confirm it explicitly."* That is the operator's to settle, not CC's.
|
||||
|
||||
## 10. Deployed
|
||||
|
||||
- Controller **0.194.0** on demo-hp guest 9201 — `Up (healthy)`.
|
||||
- Hub **0.90.1** — ArgoCD `Synced` / `Healthy`, image `gitea.dooplex.hu/admin/felhom-hub:0.90.1`,
|
||||
startup log clean (`Listening on :8080`, deadline-check scheduled for 05:00 CEST).
|
||||
|
||||
## 11. Teardown
|
||||
|
||||
Fill file removed; guest helper scripts and the credential file `shred`-ed; `pct fstrim 9201` returned
|
||||
63.3 GiB; thin pool **30.76**; the guest is back to 1.9 G used / 64 G free. The scratch email renderer
|
||||
under `hub/tmprender/` was deleted after use. The hub DB copy is scratch-only.
|
||||
|
||||
## 12. Observations — NOT acted on
|
||||
|
||||
1. **Only two apps are deployed on the demo box**, so "several apps" was 2 of 2 — exactly the measured
|
||||
case, but a five-app demonstration would be stronger.
|
||||
2. **`FormatOperatorEmail` puts an emoji in every operator subject.** Part 3 asked for none; the icon
|
||||
comes from the shared formatter and changing it globally would alter every other operator mail and
|
||||
its tests. The digest's own copy has no emoji.
|
||||
3. **A `recorded` row is written for record-only events on the `operator` channel.** It is not an
|
||||
operator *notification*, so the channel label is slightly overloaded — readable today, worth a
|
||||
thought if a third channel ever appears.
|
||||
4. **The customer-channel `skipped/operator_only` row no longer appears for the per-app event**,
|
||||
because record-only returns before the customer leg. That is correct, but anyone grepping history
|
||||
for that pattern will see it stop on 2026-08-03.
|
||||
- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row.
|
||||
- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific.
|
||||
- **R-185 / R-186 / R-187 → filed** (grep established 185–187 free; R-184 was the highest in use).
|
||||
- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself
|
||||
proposed** — which was the never-fires version.
|
||||
- `STATUS.md` rewritten for the operator, trimmed back to one screen.
|
||||
- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test
|
||||
row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key)
|
||||
and S-18 (ep0).
|
||||
|
||||
@@ -29,6 +29,14 @@ Proven end to end on real hardware.
|
||||
|
||||
## What shipped recently
|
||||
|
||||
- **Each backup is now proved, instead of the clock being obeyed.** The machine used to re-test a
|
||||
backup every twenty-four hours on a timer that restarted whenever the software was updated, so the
|
||||
test happened at an arbitrary time and a fresh backup could sit unproven while an old one was
|
||||
re-checked. Now a copy is tested once, about a day after it is made, and not again until there is a
|
||||
newer one — so the daily copy is proved daily and the weekly off-site copy weekly, each on its own.
|
||||
The alert that says "this copy has not been proved lately" learned each copy's own rhythm in the
|
||||
same change; without that it would have started complaining every night about a system that is
|
||||
working. *(R-86)*
|
||||
- **A failed backup now tells you about every app, once.** When several apps fail in one night you get
|
||||
**one** email listing all of them, what kind of backup failed for each, why, and how much space is
|
||||
left — instead of an email about one app and silence about the rest. Measured before the fix: nine
|
||||
@@ -42,21 +50,14 @@ Proven end to end on real hardware.
|
||||
refuses a release nobody can download. *(R-110, R-115)*
|
||||
- **The reserve now guards the step that fills the disk, and its promise is true.** It decides once
|
||||
per app, before writing anything, and no longer waves an app through at 96% full. *(R-181)*
|
||||
- **The backup partition is gone**; both demo machines were rebuilt on the new shape by two different
|
||||
routes. Backup space went from 19 GB to 65 GB on the small machine, 45 GB to 233 GB on the big one.
|
||||
*(R-165, R-178)*
|
||||
- **The last of the three apps that never saved their data is fixed.** *(R-156)*
|
||||
- **A filling disk warns the customer before anything breaks**, naming the drive and the space left.
|
||||
*(R-167, R-158)*
|
||||
- **The checks have two nets** — one before every push, one on the build server that emails you.
|
||||
*(R-29, R-161, R-168)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** nothing outstanding. Everything raised in the last three days is closed.
|
||||
- **Next:** restore-tests scheduled around each backup rather than on a free-running timer — now
|
||||
unblocked, because the off-site endpoint has the memory for it. *(R-86)*
|
||||
- **After:** the off-site copy that the machine making it can still erase. *(R-95, R-87)*
|
||||
- **Now:** nothing outstanding.
|
||||
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
||||
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
||||
*(R-87)*
|
||||
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
@@ -65,19 +66,20 @@ Proven end to end on real hardware.
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
machines to install, only which ones exist. Closing that needs either a password given to the build
|
||||
server or a check inside the hub itself. *(R-184)*
|
||||
- **One older question still standing:** whether the off-site endpoint counts as protected. It was
|
||||
never named either way, and it is yours to settle. *(target-selection.md)*
|
||||
- **Nothing else.** The question about whether the off-site endpoint counts as protected is
|
||||
**settled — it does**, and the machine list now says so instead of asking.
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-03** — Backups are now proved one at a time, each about a day after it is made, instead of
|
||||
on a timer; the "not proved lately" alert learned each copy's own rhythm so it does not complain
|
||||
about a healthy weekly copy. You settled that the off-site endpoint is protected, and the machine
|
||||
list records it. One thing found while testing: on the small demo machine the agent cannot see its
|
||||
own local backups at all — a permission that was never granted — so that copy has never been
|
||||
test-restored there. Written down, not yet fixed. *(R-86, R-185)*
|
||||
- **2026-08-03** — A failed backup now reports every app in one email, and nothing is dropped without
|
||||
a record. You rescaled the off-site endpoint to twice the memory, which closed one item and
|
||||
unblocked another; the documents describing the old machine were corrected, and the audit records
|
||||
of what it used to be were annotated rather than rewritten.
|
||||
- **2026-08-03** — Publishing became deliberate for both the installer and the agent, with a way back
|
||||
and a check that a release can actually be downloaded.
|
||||
- **2026-08-03** — The reserve was fixed to guard the step that fills the disk; the last unsaved app
|
||||
was fixed; both demo machines were rebuilt on the merged disk layout and proven end to end.
|
||||
- **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were
|
||||
failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this
|
||||
machine and the tester's box are protected, every other box may be broken or reinstalled freely.
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
|
||||
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). |
|
||||
| **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** |
|
||||
| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier and a failure is heard** | agent v0.104.0, hub v0.77.0 | **IMPLEMENTED** (not PROVEN-LIVE) | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why NOT PROVEN-LIVE:** rotation has not been observed selecting both tiers across consecutive UNATTENDED cadences. At a 24h cadence that is a multi-day observation window, and it had not elapsed when this shipped. A single passing run proves the code path, not the rotation. → **R-85** |
|
||||
| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **Known gap on this box, filed as R-185 and NOT caused by this change:** the agent's PVE token has no ACL on `/storage/felhom-backup`, so the HOST tier's content listing returns empty and that tier is not restore-testable there — the offsite tier is what this row's live proof covers. Evidence: `felhom-agent/REPORT.md` |
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below |
|
||||
| **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run |
|
||||
| Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 |
|
||||
|
||||
@@ -132,6 +132,41 @@ executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by const
|
||||
(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
|
||||
(INV Part D1).
|
||||
|
||||
### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03)
|
||||
|
||||
**[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to
|
||||
fire on an interval started at daemon start. It no longer does. The rule is:
|
||||
|
||||
> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
|
||||
> The tier is **DUE** when **A** exists and **A has not already been proven**.
|
||||
|
||||
So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather
|
||||
than the process's uptime:
|
||||
|
||||
| tier rhythm | what is proved, and when |
|
||||
|---|---|
|
||||
| daily (host tier) | yesterday's archive, once a day |
|
||||
| weekly (offsite tier) | last week's archive, once a week |
|
||||
| newborn (no archive yet) | nothing — **UNKNOWN, never a fault** |
|
||||
|
||||
**The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest
|
||||
archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive
|
||||
age to zero long before it reaches the lag — so the literal reading silently switches restore-testing
|
||||
off for the tier that matters most.
|
||||
|
||||
What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its
|
||||
journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on
|
||||
success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker
|
||||
remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one
|
||||
due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN).
|
||||
|
||||
**The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very
|
||||
cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval
|
||||
just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay
|
||||
tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed
|
||||
archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite
|
||||
retention), and falling back to the tier's declared rhythm when history is too short to observe one.
|
||||
|
||||
### Why the split is right, stated once
|
||||
|
||||
**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
|
||||
@@ -820,7 +855,7 @@ does **not** hold as written. → **R-108**
|
||||
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
|
||||
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
|
||||
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
|
||||
| R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written |
|
||||
| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
|
||||
| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
|
||||
|
||||
### 10.3 Divergences that are documented elsewhere and are not re-opened here
|
||||
@@ -886,7 +921,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
|
||||
| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
|
||||
| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
|
||||
| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
|
||||
| unattended restore-test across tiers | **IMPLEMENTED** (rotation not observed across consecutive cadences) | `00-capability-map.md:41`; LIVE per-tier timestamps this session |
|
||||
| unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) |
|
||||
| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
|
||||
| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
|
||||
| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
|
||||
|
||||
@@ -26,8 +26,11 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** With the target absent the page rendered the ABSENT copy (1), the system-disk copy 0, the offer block 0 — both of E-2d's falsehoods gone. API carried `message:"A rendszermentés meghajtója nem érhető el…"` with `target:felhom-backup`. **FIXED: the third state exists.** New `BackupTargetState.TargetAbsent` separates *configured-and-gone* from *never-configured*. `Degraded` keeps its meaning (is there a problem) so the wire contract is unchanged for every consumer; `TargetAbsent` answers which problem, because the remedies are OPPOSITE — attach any second drive vs reconnect *that* one. Copy routed through `degradedMessageFor` (still one decision point) and taken **verbatim** from the hub's `backup_target_absent` email so banner and mail tell one story. **Offer suppressed on the branch itself**, deliberately not left to `firstOfferableDrive`'s `Disconnected` skip — that flag is set by R-113 in another repo, and this state must be right without it. Red-proof: deleting the branch reproduces E-2d's exact payload, offering `/mnt/felhom-drives/mentes2`, the drive that had vanished. **MinAgent unchanged 0.113.0** — R-114 reads `BackupTarget`/`MountPath`/`GuestPath`/`Role`, none of which R-113 altered, so demo-hp is not held. **NOT live-validated: Scenario C cannot occur on a healthy box.** `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC |
|
||||
| **R-29** | **The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it.** This is the **class**, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. `controller/scripts/docker_run_volume_path_gate.py` failed continuously from **2026-07-14 (v0.129.0)** until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" | **CLOSED — both halves shipped** (2026-08-02) | — | **This item has existed at `ROADMAP.md:158` since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding**, because it is an open item *about work not getting done* that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: **(a)** the `docker_run_volume_path_gate` finding is benign and the fix is a 3-line ALLOWLIST addition with its why — **not** a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; **(b)** the systemic half, the real item: decide where gates run (pre-push hook, `build.sh` step, or CI) and make a red gate block the train the way the Go green gate does. **Two further orphans confirmed 2026-07-29** by repo-wide grep across all file types + sibling repos + `~/.claude` settings/skills/hooks + `.git/hooks` (none non-sample) + Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) + CI-directory find (**this repo has no CI at all**) — every one of the 19 hits is a docstring, a code comment or prose, and **not one is an invocation**: `scripts/hostinstall_gates.py` — **RED today** (`hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1), the same finding as **R-94 leg (b)** — and `scripts/hub_confirm_gate.py`. Of the four gates in `scripts/`, only `site_gates.py` is mandated anywhere (`CLAUDE.md:153`) and `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance** — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either **UPDATE 2026-08-02 — leg (a) CLOSED** (`felhom-controller` `c432f70`, its own reviewed diff as specified): `appexport/estimate.go`'s `-v` is a NAMED VOLUME mounted read-only into a throwaway container, no host path, structurally identical to the allowlisted `backup/backup.go` entry — allowlisted with its why; `realVolumeSize` untouched. **Leg (b) HALF-SHIPPED:** the 'decide where gates run' ruling is now made and half-implemented — **every repo has ONE entry point** (`felhom.eu/scripts/repo_gates.py`, `felhom-controller/controller/scripts/controller_gates.py`, `felhom-agent/scripts/agent_gates.py`, `app-catalog-felhom.eu/scripts/catalog_gates.py`), each mandated in its `CLAUDE.md` and each wired to `.githooks/pre-push` via `--fast`. **THE CENSUS, which is the finding:** thirteen gate scripts across four repos; **every gate a `CLAUDE.md` names was GREEN, and two of the four nobody names were RED** — `hostinstall_gates.py` (red since 2026-07-14) and `reuse_refs_check.py` (red on all four repos); a third, `docker_run_volume_path_gate.py`, was named only in `REUSE.md:284` and was also red. Correlation with 'named in a CLAUDE.md' was exact. **STAYS OPEN for the automatic half** — a hook is per-clone and `--no-verify` skips it; the unbypassable half is CI → **R-168** **CLOSED 2026-08-02, on the demonstrated ALARM and not on a green run.** Leg (b)'s automatic half is now live: a Gitea Actions runner re-runs every repo's entry point on every push, independent of who pushed and of what they typed (→ R-168). The class this row opened — *a gate that exists, asserts something true, is red, and is invoked by nothing* — is answered at both ends: the pre-push hook refuses locally, and CI catches a `--no-verify` bypass and **emails the operator**, proven with a real red run and a provider accepted-id. What remains is not this row's finding but a working-style choice — CI reports rather than blocks because there is no merge to gate (→ R-169) | CC |
|
||||
| **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC |
|
||||
| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY (M) — UNBLOCKED and re-ranked 2026-08-03** | — | Trigger a tier ~24 h after **its own** newest archive. **The R-90 dependency is discharged:** that row informed the cadence because ep0 had 3.8 GB and a 14.46 GB restore read had OOMed it, so a more frequent restore-test risked knocking the offsite endpoint over. ep0 is now a **CX33 with 8 GB RAM plus a 4 GiB swapfile** (measured 2026-08-03), so headroom is no longer what sets the cadence and this can be designed on its own merits. **Do not read that as "the ceiling is gone":** the OOM was a 14.46 GB restore against 3.8 GB, and 8 GB is comfortable rather than unbounded — the restore-test cadence should still be paced, just not by fear of the endpoint | CC |
|
||||
| **R-87** | The restic tier is never restore-tested | **READY** | — | Design a controller-side test (no scratch-guest analogue transfers) | CC |
|
||||
| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.121.0**, hub **v0.91.0**) | — | **The rule that shipped:** *let A be the newest archive on a tier that has settled ≥24 h; the tier is DUE when A exists and A has not already been proven.* Daily tier → proved daily on yesterday's archive; weekly tier → weekly on its own; newborn → UNKNOWN. The daemon-start ticker survives only as the **evaluation interval**. **THE TRAP, recorded because it is the version a reasonable person writes:** the row's own wording implemented literally — *"due when the newest archive is ≥24 h old"* — is NEVER true on a **daily** tier, because a new archive resets the newest-archive age to zero long before it reaches the lag; it would have silently switched restore-testing OFF for the tier that matters most. Red-proved at **0 runs over 5 simulated days**. **The state now records WHICH archive was proven**, not when a tier last passed — a time cannot answer *have we proven this archive*. A pre-R-86 state file keeps its time (rotation ordering survives) and yields no proven archive, so each tier is due exactly once after the upgrade: the safe direction. **Two knobs replace one and the old one is not silently repurposed:** `restore_test_eval_interval_seconds` (6 h) and `restore_test_settle_seconds` (24 h); the deprecated `restore_test_cadence_seconds` keeps its DISABLE meaning verbatim, now seeds the settle lag, and the daemon WARNs once at start-up naming both. **6 h is bounded from both ends, not picked:** MEASURED cost of one evaluation on demo-felhom — local dir storage **18 ms**, PBS tier over the WAN to ep0 **392 ms**, both **430 ms** — so cost is irrelevant; the CEILING is that a FAILING tier stays due, making the evaluation interval its retry interval for a multi-GB restore. **Part 2 shipped WITH it and was not optional** — see the hub half in this row's sibling text and `07-backup-architecture.md` §3: `restoreProvenStaleAfter` was a flat 7 days derived from the very cadence this removed, and a healthy weekly tier's proof age reaches **exactly** 168 h against a 168 h window — it sat ON the line, so any ordinary delay tipped it into a nightly alarm about a working system. The window is now per tier from that tier's observed archive interval, ×4 generations, floored at the old 7 days and capped at 12 days (strictly inside the two-week offsite retention), falling back to the tier's DECLARED rhythm (26 h host / 8 d offsite — the backup-freshness checker's own thresholds) when history is too short to observe one. **A hollow test caught by its own red-proof:** the first Scenario-G fixture had no jitter and PASSED under the flat-window mutation, because a perfectly regular weekly tier sits exactly ON the line rather than over it. The jitter is what makes it a test. **Also fixed in passing:** the candidate picker now skips archives failing `archivePlausiblyComplete` (under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof and leave the tier due at EVERY evaluation), and the due-check runs BEFORE the heavy-operation gate is taken (a frequent poll must not be able to make a starting backup record a failure — F-A1). **Live proof:** see `felhom-agent/REPORT.md` | CC |
|
||||
| **R-87** | The restic tier is never restore-tested | **READY — RE-RANKED UP 2026-08-03 (R-86 closed)** | — | Design a controller-side test (no scratch-guest analogue transfers). **Most of what this row needed now exists.** R-86 built the piece that was missing: a tier is proved **per archive generation**, on its own rhythm, with the proof recorded as *which archive* — which is exactly the shape a weekly-ish restic tier needs, and the reason this row could not simply reuse the whole-guest scheduler before. What remains is genuinely restic-specific and is NOT a scheduling problem: there is no scratch-guest analogue, so the test has to be a controller-side restore of a bounded sample into a throwaway path, with its own definition of "proved". **Two things to carry over rather than re-derive:** the proof must record the SNAPSHOT it proved (not a timestamp), and the hub's staleness window must learn this tier's rhythm the way `restoreProvenWindow` now does — a restic tier on a weekly cadence lands on the same false-alarm line the flat 7 days did. **And R-95 still applies:** that credential can delete, so a restic restore-test must never be able to write to the repo | CC |
|
||||
| **R-185** | **The agent cannot see the host backup tier's archives on demo-felhom — the PVE token has no ACL on `/storage/felhom-backup`, so the content listing returns EMPTY where root sees three archives.** Found 2026-08-03 while live-validating R-86. `pveum acl list` grants `FelhomAgentStore` on `/storage/{local,local-lvm,felhom-pbs}` and **not** on `felhom-backup`, which is the box's actual `local_backup_target`. Verified three ways: `pvesh` as root lists 3 archives (6.1–6.3 GB, 08-01/02/03); the same endpoint with the agent's token returns `{"data":[]}`; and `local` — which HAS a grant — returns its archives through the same token | **OPEN — filed, not fixed** | — | **Pre-existing and independent of R-86** (it is a property of the ACL, and the R-85 rotation had the same blindness). **Consequences:** the host tier has never been restore-testable on that box, and R-85's *"an empty tier is skipped, not failed"* rule made that silent. **The part worth fixing is the silence, not only the grant:** a permission-blinded tier is today INDISTINGUISHABLE from a newborn one — both report *"no settled archive yet"* — which is this project's own absence-is-not-evidence rule failing in a new place. The agent already knows better: it RECORDS successful backups to that target, so *"I wrote archives here and the tier lists none"* is a contradiction it can detect and should say loudly. **Do not fix by widening the token blind:** decide whether the host-install ACL set should follow `local_backup_target` (it currently hardcodes `local`), which is where the drift began | CC |
|
||||
| **R-186** | **A released agent binary's sha256 cannot be reproduced from its tag.** `release-agent.sh` builds at step 3 and tags at step 4, so Go's VCS stamp records a PSEUDO-version (`v0.120.1-0.20260803130452-4d825910…`) in the published bytes, while any rebuild after the tag exists stamps `v0.121.0` — a different binary. Measured 2026-08-03 on v0.121.0: published `b2128f3c…` (14 081 336 B) vs rebuild-at-tag `8302e396…` (14 077 240 B), identical source, identical toolchain, 4 096 bytes apart | **OPEN** | — | **Why it matters:** the sha the operator vouches is the one thing tying a machine to a binary, and today nobody can independently rebuild it to check. **The build order is deliberate** (the script's own comment: a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it), so the fix is not to swap the steps blind. Candidates: `-buildvcs=false` or `-trimpath` for a version-stable stamp, or tag-then-build with the tag deleted on a failed publish. **Mitigation used this session:** the DEPLOYED binary is the PUBLISHED artifact, downloaded from Gitea — not a local rebuild — so the running bytes are the vouchable ones | CC |
|
||||
| **R-187** | **R-115's one-command release had never actually run its publish leg — the first real use died there.** `scripts/publish-agent.sh` has been mode `0644` since it was created (2026-06-28), because every earlier caller invoked it as `bash scripts/publish-agent.sh`; `release-agent.sh` (written 2026-08-03) called it directly and got `Permission denied` on v0.121.0's release | **CLOSED — SHIPPED 2026-08-03** (`felhom-agent`) | — | **Fixed both ways in one commit:** the executable bit restored, and the caller changed to `bash "$REPO_ROOT/scripts/publish-agent.sh"` so the release no longer depends on a file mode — the kind of thing a checkout, an archive or a copy silently loses again. **The lesson is R-115's own, one level up:** the mechanism written to make a step unforgettable was itself never exercised end-to-end, so it failed the first time it mattered. A mechanism that has not been RUN is a note with better formatting | CC |
|
||||
| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
|
||||
| — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator |
|
||||
| **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |
|
||||
@@ -129,9 +132,15 @@ there is one ranking to maintain rather than two.
|
||||
already fetches 1.22.0. What remains is a wrong label plus two pieces of dead safety equipment —
|
||||
a drift gate nobody runs and a test that compares a constant to itself. Cheap and worth doing;
|
||||
not high-consequence, and it blocks nothing.
|
||||
3. **R-86** — an operator ruling already exists; it only waits on knowing what load ep0 can take.
|
||||
4. **R-87** — real and unbuilt, but needs its own design, so it should not jump work that is specified.
|
||||
5. **R-110** — last **because it is not a READY row**: the ruling is the operator's, not CC's, and
|
||||
3. ~~**R-86**~~ — **CLOSED 2026-08-03**, agent v0.121.0 + hub v0.91.0, proven live on demo-felhom.
|
||||
4. **R-87** — **re-ranked UP**: R-86 built most of what it was waiting for (per-archive due-ness, a
|
||||
proof that names its archive, and a staleness window that learns a tier's rhythm). What is left is
|
||||
restic-specific — there is no scratch-guest analogue — so it still needs its own design, but it is
|
||||
no longer waiting on a scheduling model that did not exist.
|
||||
5. **R-185** — the agent is blind to demo-felhom's host backup tier (a missing storage ACL), and the
|
||||
blindness reads exactly like a newborn tier. Small to fix, and the *silence* is the part worth
|
||||
fixing, not just the grant.
|
||||
6. **R-110** — last **because it is not a READY row**: the ruling is the operator's, not CC's, and
|
||||
there is nothing for CC to build until it lands. Ranked here rather than omitted because it is the
|
||||
only item on this page about the *publish channel* of the most privileged artifact Felhom ships,
|
||||
and today's exposure is zero — which makes now the cheapest moment it will ever be to decide.
|
||||
|
||||
@@ -144,7 +144,7 @@
|
||||
| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
|
||||
| R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. |
|
||||
| R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
|
||||
| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — UNBLOCKED 2026-08-03 (R-90 closed: ep0 is now a CX33, 8 GB + 4 GiB swap, so endpoint headroom no longer sets the cadence — though 8 GB is comfortable, not unbounded: the original OOM was a 14.46 GB restore)** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
|
||||
| R-86 | ~~**Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup**~~ | S | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03 (agent v0.121.0 + hub v0.91.0)** | **The reasoning that produced the row still holds and is worth keeping:** R-85 scheduled on a free-running INTERVAL with oldest-proven-first rotation, which cannot express *"the day after the PBS backup"* — any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. The operator's ruling was weekly per tier, ~1 day after that tier produced a new backup, so the gap between an archive being written and being proven stays small and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **What shipped is that ruling, with one correction the row's own wording would have got wrong:** the shape written here — *"test tier X once its newest archive is ~24 h old and not yet proven"* — is NEVER true on a DAILY tier, because a new archive resets the newest-archive age to zero before it reaches 24 h. The rule that works is *the newest archive that HAS settled 24 h, and has not been proven*: `A = newest archive older than the settle lag`, due iff A exists and A ≠ the proven archive. **The interim is retired:** `restore_test_cadence_seconds = 302400` (3.5 d) is no longer the mechanism; that key now only disables (negative) or seeds the settle lag. Alignment is no longer approximated by a cadence — it is the schedule. Full detail: `OPEN-ITEMS.md` R-86, `07-backup-architecture.md` §3 |
|
||||
| R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
|
||||
| R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | **Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change)** — **Part 1** added the failure breaker: consecutive failures tracked per TARGET, backoff `15m→30m→1h→2h→4h` capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to `TriggerNow`, and `stillRunning` is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. **Part 2 is agent-side and needs a wire change** — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user