From b7c5cfc061884a17a7dd596ba2fca0bbac1ca504 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 4 Aug 2026 11:12:08 +0200 Subject: [PATCH] docs: R-193 spike report, operator STATUS, and the ROADMAP R-1 scope correction (F3 covers HOST re-enroll, not a guest rebuild) --- REPORT.md | 520 +++++++++++++++++-------------- STATUS.md | 56 +++- documentation/backlog/ROADMAP.md | 2 +- 3 files changed, 342 insertions(+), 236 deletions(-) diff --git a/REPORT.md b/REPORT.md index a52861c..e981156 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,263 +1,325 @@ -# REPORT — R-191 (a backup that worked must not report failure) + R-190 (a permission must repair itself) +# REPORT — R-193/R-192 spike (no code) + R-195 phantom-customer alarm (hub v0.92.0) -**Date:** 2026-08-04 · **Repos:** `felhom-agent` **v0.123.0 → v0.124.0 → v0.124.1** (`257c4d8`, and -the v0.124.1 fix) · `felhom.eu` installer **1.24.0 → 1.25.0** (tag `installer-v1.25.0`) · -**no hub change and no hub bump**. +**Date:** 2026-08-04 · **Class:** spike (Parts 0–3) + one fenced implementation (Part 4) +**Boundary held:** the spike shipped **no production code for R-193 or R-192**, and Part 4 never +touched the spike's subject matter. + +> Supersedes the earlier 2026-08-04 report (R-191 + R-190), per the overwrite convention. Nothing +> durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `CONTEXT.md` S-24/S-25 and +> `OPEN-ITEMS.md` R-190/R-191. --- -## 1. Baselines +## 1. Confirmed baselines, re-read at the cited commit -| Repo | cited in §1 | found on arrival | matched? | +| Repo | `main` @ commit on arrival | Version | End state | |---|---|---|---| -| `felhom-agent` | `72161f6cf010` / `v0.123.0` | same | **yes** | -| `felhom.eu` | `d692c46db08f` / installer `1.24.0` | **`33f0ab2f33f7`** / installer `1.24.0` | **version yes, commit NO** | +| `felhom.eu` | `f456835bbcc4` ✔ matches the spec | hub `v0.91.1` (deployed `felhom-hub:0.91.1`) | **`v0.92.0`** built, pushed, manifest bumped, synced, **live** | +| `felhom-controller` | `0887fd676d60` | — | **untouched** (read-only) | +| `felhom-agent` | `856a127` | — | **untouched** (read-only) | -`felhom.eu` had moved four commits forward since the brief was written — this morning's alert -diagnosis (R-190/191/192/193 rows, the demo-hp offsite re-issue), all pushed. Recorded rather than -glossed. Both `--ref=installer-v1.24.0` lines confirmed at 327 and 372. +Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`. +Highest register ID on arrival was **R-194**; grepped before minting. -## 2. Part 0 — the gate, and the three broken instruments that nearly failed it +--- -**VERDICT: PASS.** ep0 prunes **both** namespaces on schedule. +## 2. Q1–Q8 — each answered or explicitly marked unmeasured + +Full method, quotes and query output: **`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md`**. + +### Q2 and Q3 first — they decide whether the rest matters + +**Q2 — does a rebuilt controller reuse the escrowed repository password, or mint a new one?** +**RULING: it MINTS A NEW ONE.** +*Method — source:* `ConfigureOffbox` → `ApplyOffsiteTarget` → `WriteOffboxSecrets`, which generates a +fresh 256-bit password whenever `/offbox/repo_password` is absent +(`felhom-controller/controller/internal/backup/offbox.go:392-401`). A rebuilt guest has a fresh data +dir. The only recovery path, `InjectOffboxPassword`, has **exactly one caller in the entire repo** +(census: `grep -rn "InjectOffboxPassword" --include=*.go .` → 3 hits, 1 definition, 1 caller, 1 +comment) — `web/offbox_handlers.go:189`, a web form a human pastes into. **The apply-bridge never +calls it.** +*Method — live, without touching a box:* `host_escrow.restic_pw_sha256` vs +`host_escrow_superseded.restic_pw_sha256`. **demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom +`48741892…` → `c60c8bc7…`. Both CHANGED.** +**This is bigger than R-193 as filed:** demo-felhom, which R-193 records as having survived by luck, +had **36 snapshots / 1,136,685,919 B** before its rebuild and has reported `snapshot_count: 0` in all +109 reports since. Its luck restored **delivery**, not the **repository**. + +**Q3 — what did the 04:15 run report?** **UNMEASURED, and the binary in the question is wrong.** +The run is daily at ~02:15 UTC (measured `last_run` values on both boxes). Neither box could run on +2026-08-04: demo-hp had no target at all until 07:15:47 UTC; demo-felhom's target was +`escrow_state: pending` from 08-03 07:19:10 to 08-04 07:20:13, which `OffboxRunnable` blocks +(configured **AND** escrowed). **The decisive run is 2026-08-05 ~02:15 UTC.** Source predicts a +**third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`; +`repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies +**orphaned** → `ErrOffboxOrphaned` → **the run refuses and shows the orphan card.** Recorded as a +prediction, not a measurement. + +### The rest + +- **Q1 — what is one-shot.** Full table in the spike. Headline: the hub's one-time provider password + is one-shot **but freely re-issuable**; the box's SFTP key is **regenerated on every apply by + design**; the **restic repository password** is the one nothing can restage — the agent's own source + calls it *"irreplaceable (unlike the SFTP access key…)"* + (`felhom-agent/internal/escrow/identity.go:35-39`). **R-193 was filed against the recoverable + secret; the damage is in the irreplaceable one.** +- **Q4 — see §3 below.** +- **Q5 — why one box recovered and the other not. CONFIRMED FROM THE LEDGER.** demo-felhom's + `one_time_secrets` row was created `2026-07-21 08:29:29` (traceable to a Re-issue at that exact + timestamp) and sat **UNCONSUMED for 13 days** until the rebuild consumed it at + `2026-08-03 07:17:58`. Without one: exactly demo-hp — `consume` → **404** and a retry loop, 25 hours + unprotected. **Correction to the contrast:** the recovery was of the plumbing only (Q2). +- **Q6 — why delivery is one-shot.** The design's own reasons quoted from three sites. **The security + half, stated:** the consume endpoint is gated by `checkAuthCustomer` — **the customer's API key, the + same credential as the config pull** — so an automatic restage grants **no new access to an + unauthenticated party**. What it genuinely widens is the **exposure window**: a live provider + password becomes fetchable whenever the box reports no offsite target, rather than only in the + seconds after an operator acts. Blast radius stays the customer's own label-scoped sub-account. The + R-39(a) unconsumed-secret guard must survive any change. +- **Q7 — R-192's guard. BOTH HALVES CONFIRMED WITH NUMBERS.** See §4. +- **Q8 — what each candidate costs.** See §5. + +--- + +## 3. Which of the two contradictory claims about Re-issue is true + +**The claim that Re-issue "only resets the transport credential" is TRUE about what +`ReissueCredentials` does. R-39's "rotates the restic password" is FALSE.** + +From source (`hub/internal/offsite/offsite.go:150-228`), the function does four things: `genPassword`, +`ResetSubaccountPassword`/`ResetBoxPassword`, `SaveOneTimeSecret`, and bookkeeping. **There is no +reference to a restic password anywhere in it, and none is possible** — that password is generated on +the box and never leaves it except into the R-wrapped escrow, which the hub cannot open. + +**But the bookkeeping is justified by the false premise**, `offsite.go:198-201`: *"the restic repo +password just changed, so any existing key-escrow blob … is now STALE"*. The same wrong claim appears +at `api/handler.go:1067-1069` and in R-193's own row. **Eighth entry in `CLAUDE.md`'s table of +comments asserting a guarantee the code does not provide** — the first where the comment factually +describes a *different function*. → **R-196**. + +**What corroborated it, and the corroboration is the strongest part.** The task asked: if the repo +password was untouched, what explains demo-hp's escrow going `pending` after the Re-issue? **Fully +explained, and not by the Re-issue:** `ApplyOffsiteTarget` carries `EscrowState` over from the +*existing* target, the rebuilt guest had none, so the `pending` fallthrough fired — and the box had +just minted a new repo password, so `pending` was *correct for a different reason*. +**The proof they are independent is demo-felhom:** it received **no Re-issue at all** (its only +`escrow_stale`/`offsite_reissued` pair is dated `2026-07-21 08:29:29`), its repo password changed +anyway, and **nothing marked its escrow stale for 13 hours.** The mechanism is exactly inverted: +`escrow_stale` is wired to the one path that does **not** change the repo password and absent from the +path that does. + +--- + +## 4. Every query used, with its fields and values + +The DB snapshot was taken from the live pod and its **freshness proven with a positive observable**, +not with "the query returned no error": + +| # | Query / read | Result | +|---|---|---| +| 1 | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale (mtime 08:34 vs wal 10:45 CEST). Copying only the main file is the R-3 shape. | +| 2 | `PRAGMA integrity_check` | `ok` | +| 3 | `MAX(host_reports.received_at)` vs `datetime('now')` | `2026-08-04 08:45:38` vs `08:49:46` — **newest row 4 min old** | +| 4 | `SELECT customer_id, status, created_at FROM customer_configs` | `demo-felhom` active · `peti-felhom` active · `demo-hp` active · `drill-r50` **blocked** · `david` active (created `2026-08-01 16:51:49`). **`sess-f` is gone — confirmed, not assumed.** `drill-r50` untouched. | +| 5 | `hosts` / `host_deletions` / `host_reports` / `reports` counts for `david` | **0 / 0 / 0 / 0** — never bound | +| 6 | `events WHERE event_type='expected_dbdump_missed' AND created_at > '2026-07-25'` | **only `david`**, 03:00 UTC on 08-02, 08-03, 08-04 | +| 7 | `SELECT customer_id, MAX(received_at), COUNT(*) FROM reports GROUP BY customer_id` | `demo-hp` 1329 · `demo-felhom` 6690 · `drill-r50` 183 (last 07-29) · `peti-felhom` 482 (last **2026-07-15**) — the reason peti-felhom is `down` and does not alarm | +| 8 | distinct `$.offsite` objects per box with first/last seen | demo-hp: `15 snapshots / 40 927 044 B` until `08-03 05:59:21`, **ABSENT** `06:12:19`→`08-04 07:12:01` (111 reports), then `pending` 07:15:47, `escrowed` 07:16:02, `snapshot_count 0`. demo-felhom: `36 snapshots / 1 136 685 919 B` until `08-03 07:01:28`, ABSENT ×1, then `pending` `07:19:10`→`08-04 07:20:13`, `escrowed` 07:20:28, `snapshot_count 0` | +| 9 | `host_escrow` + `host_escrow_superseded` `restic_pw_sha256` | **the decisive measurement** — both changed (§2) | +| 10 | `one_time_secrets` (no values read) | demo-felhom created `2026-07-21 08:29:29`, consumed `2026-08-03 07:17:58`; demo-hp created+consumed `2026-08-04 07:11:51`/`07:12:06` | +| 11 | `$.claimed` on the latest report of each box | **`1` on both** — which selects `ensureOffboxRepo`'s refuse branch | +| 12 | `CountReportsOffsiteSince` reproduced verbatim (`ORDER BY id LIMIT 500`) for demo-hp @ `2026-07-23 09:53:41` | **`total=500, withOffsite=500`**, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** | +| 13 | same window, **no LIMIT** | `total=1174, withOffsite=1063` (⇒ 111 without — matches #8) | +| 14 | full text of the live `offsite_delivery_stuck` message | *"…and **500 report(s) since carry no offbox target**…"* while `OffsiteReportsSinceConsume` was **500** | +| 15 | `COUNT(*)` of `offsite_credential_restaged` | **0, for every customer, ever** | +| 16 | the shipped predicate run verbatim against live data (post-deploy) | `david` → **0**; `demo-felhom`/`demo-hp`/`drill-r50`/`peti-felhom` → **1** | + +**Instrument limits recorded rather than discovered later:** `one_time_secrets` has `customer_id` as +PRIMARY KEY and is last-write-wins **by design**, so it holds only the newest credential per customer — +demo-hp's `2026-07-23` consume, on which R-192's whole diagnosis rests, was **overwritten** by +yesterday's Re-issue and is no longer readable from that table. The `2026-07-23 09:53:41` anchor in +query #12 comes from R-192's record, and that is stated in the spike rather than presented as a live +read. + +--- + +## 5. Ranked options, with a recommendation + +| rank | option | verdict | +|---|---|---| +| **1** | **(d) the honesty pass — now** | Cheap, weakens nothing, and **the only option that helps the incident already in flight**. Two parts: correct R-192's message + log its refusal (**R-196**), and **detect a changed `restic_pw_sha256`** — both hashes are already in the hub's DB and **nothing reads them** (**R-197**). | +| **2** | **(c) agent-retained repo password — NOT PREVIOUSLY NAMED** | The only candidate aimed at the irreplaceable secret. The agent lives on the Proxmox host, **survives a guest rebuild**, already receives the password over the pinned local API and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Every seam exists. Price: a copy of the data key at rest on the host. **The operator's trade to make.** | +| **3** | **(a) hub restages automatically** | **Already exists** as `reissueOnReenroll`'s F3 leg, wired to the wrong event (behind `handleHostEnroll`'s mint-once-reuse short-circuit; a guest rebuild leaves the `hosts` row intact). Worth doing **after** (c), never instead: on its own it restores the transport to a box whose data key is already gone, and **would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** — strictly worse than the current loud failure. | +| **4** | **(b) recover from escrow at re-bootstrap** | **Not implementable as stated.** The escrow is R-wrapped and zero-knowledge; the hub has no recovery code. "Recoverable at re-bootstrap" describes a customer-present ceremony — i.e. the manual form that already exists. | + +**Recommendation: ship (d) now; then decide (c). Do not ship (a) first.** + +### The operator's question — stated, and NOT answered + +> The irreplaceable secret is the restic repository password. It is generated on the box, destroyed by +> a guest rebuild, and the only surviving copy is sealed under the customer's recovery code. Every +> machine will be reinstalled. +> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, +> where you already have root — or do you want it to stay sealed and accept that every rebuild starts +> a new repository, provided the system says so loudly on the day it happens?** + +**STOPPED here. Neither candidate was begun.** + +--- + +## 6. Part 4 — the phantom-customer alarm (SHIPPED) + +**Files changed** +- `hub/internal/store/store.go` — **new** `(*Store).HasEverBoundHost(customerID) (bool, error)`: + a live `hosts` row **OR** a `host_deletions` tombstone. +- `hub/internal/monitor/deadline.go` — one guard at the top of `CheckBackupDeadlines`' customer loop, + plus an `unbound` counter in the summary line. +- `hub/internal/monitor/deadline_unbound_test.go` — **new**, 5 tests. + +**Commit:** `7fff45d688fa` (code + tests + spike + docs) · `f581ac134964` (manifest bump). + +**The mechanism, established at source and corroborated live.** The existing down-skip could never +have covered this: it reads `StalenessChecker.GetState()`, whose map is seeded from +`store.GetCustomers()` — **a query over the `reports` table** (`store.go:1055-1067`). A customer with +zero reports is in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. **The +skip misses exactly the customer it would most obviously cover.** Corroborated by query #7 and by the +live post-deploy startup log — `Staleness checker initialized: 2 ok, 0 stale, 1 down` — where the +`1 down` is `peti-felhom` (host deleted 2026-07-15, 482 old reports) and `david` is **absent from the +census entirely**. + +**The discriminator is "was a host EVER bound", not "has a report arrived"**, which is the point: +a box that was installed, bound and went silent has a `hosts` row and keeps alarming. **Fail-open on a +read error** — an unreadable binding must never suppress a real alarm. The deferral is **logged with +its own counter** (the v0.73.0 Part-7 precedent). The anchored-verdict structure is untouched, and +`david`'s record was not modified. + +**Sibling verdicts checked rather than assumed.** The census of every customer-iterating checker: +`deadline.go` backup half (already guarded by `reportJSON == ""`), `restoretest.go` (guarded by +`latest == ""`), `staleness.go` + `offsite.go` (iterate `GetCustomers()`, report-derived — a hostless +customer never appears), `offsite_delivery.go` (a hostless customer derives `DeliveryNoSecret` and is +skipped). **The DB-dump half was the only verdict that could fire**; the guard covers both verdicts in +that function uniformly. + +**Test results — the suite run and the exit code read BEFORE any commit, never in one command.** ``` -prune-demo-felhom store=felhom-offsite ns=demo-felhom schedule=03:30 keep-last 2 -prune-demo-hp store=felhom-offsite ns=demo-hp schedule=03:30 keep-last 2 +go build ./... → BUILD OK +go vet ./... → VET OK +go test ./... → rc=0, 17 packages ok, no FAIL lines ``` -**18 prunejob tasks, all `status=OK`, every day from 2026-07-27 to 2026-08-04**, two per day, one per -namespace. The newest task log, read in full: +The 13 `TestCheckBackupDeadlines*`/`TestHasEverBoundHost` tests were run with `-v` and **the `=== RUN` +lines were checked**, per `CLAUDE.md`'s rule that a `-run` pattern matching nothing prints `ok` and +exits 0. + +**THE RED-PROOF THAT MATTERS.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms` is the +case the change could break — a real box that was installed and never phoned home, indistinguishable +from `david` on every signal *except* the one the guard reads. With the guard removed it **still +passes**, which is the correct result and proves the test is aimed at the guard's *risk*, not its +presence. The companion red-proof of the fix itself, **observed**: ``` -prune job 'prune-demo-felhom' / task triggered by schedule '03:30' -retention options: --ns demo-felhom --max-depth 0 --keep-last 2 -Pruning group demo-felhom:"ct/9201" -keep ct/9201/2026-07-27T19:55:41Z -keep ct/9201/2026-07-28T04:49:43Z -TASK OK +=== RUN TestCheckBackupDeadlines_NeverBoundHost_Silent + deadline_unbound_test.go:58: a customer with NO host ever bound must raise NOTHING; + got [expected_dbdump_missed] +--- FAIL: TestCheckBackupDeadlines_NeverBoundHost_Silent ``` -Read-only throughout: no job created, edited or run; nothing deleted. ep0 is Tier 2. +— verbatim the event `david` sent three mornings running. `deadline.go` restored from a backup taken +before the mutation; full suite re-run green afterwards. -### The part that matters more than the verdict +**Deployed version.** `felhom-hub:0.92.0` built + pushed (`sha256:ca71140cf54a…`), +`manifests/hub.yaml` bumped `0.91.1 → 0.92.0` in git, ArgoCD hard-refresh + **deliberate sync** (never +`kubectl set image`). Verified: app **Synced / Healthy**, `rollout status` → `deployment "hub" +successfully rolled out` (rc=0), running image `gitea.dooplex.hu/admin/felhom-hub:0.92.0`, pod +`hub-6775889d95-gc8sk` 1/1 Running, clean startup log. -**Three of my own queries said the endpoint prunes NOTHING, and all three were broken instruments:** +**Is the 05:00 alarm confirmed silent? NO — CONFIRMATION IS PENDING, and it is stated rather than +assumed.** The hub's own log says `deadline-check: next run at 2026-08-05 05:00 CEST (in 17h53m57s)`; +there is no manual trigger (`CheckBackupDeadlines` has exactly one caller, the scheduler at +`cmd/hub/main.go:598`). An absent alarm tomorrow would in any case be weak evidence, so a **positive** +observable was taken instead: the shipped predicate was run verbatim against a fresh copy of the live +DB (query #16) — `david` → **0** (will be skipped), every other customer → **1** (behaviour +unchanged). **The 05:00 firing itself is still owed.** -| query | why it was wrong | -|---|---| -| `'prune' in t['worker-type']` → *0 prune tasks* | the field is **`worker_type`**; `worker-type` does not exist, so every task read `None` | -| `worker_type == 'prune'` → *0* | the worker type is **`prunejob`**, not `prune` | -| `journalctl -u proxmox-backup \| grep -c prune` → *0* | the unit is **`proxmox-backup-proxy`** | +--- -A fourth reading was mine to mis-frame: *"3 snapshots under keep-last 2 proves nothing is pruning"* — -the third snapshot landed at 04:49, **after** that day's 03:30 window, so it proved nothing. I caught -that one myself; the other three were caught only by checking the field names when the answer looked -too categorical. +## 7. `OPEN-ITEMS.md` rows, and the capability map -**Had I acted on them, I would have disabled the only pruning ATTEMPT while filing a finding that -nothing prunes anywhere** — a weekly false alarm traded for unbounded growth on the protected -endpoint, invisible for months. The gate did its job precisely because it demanded evidence rather -than a verdict. +- **R-192** — updated with Q7's confirmed numbers; **NOT closed**, and now carries a *new* reason not + to fix it in isolation (a successful auto-restage would have protected the plumbing while the data + was already gone). +- **R-193** — updated with the whole spike; **NOT closed**, state moved to **WAITING-ON-OPERATOR**, + owner `CC + operator`. The old "15+ vs 1 decides it" line is struck through and replaced with the + measured answer plus the third predicted outcome. +- **R-195** *(new)* — the phantom-customer alarm. **SHIPPED**, hub v0.92.0. +- **R-196** *(new)* — `escrow_stale` wired to the path that does not change the repo password, absent + from the one that does. **OPEN.** +- **R-197** *(new)* — the hub holds both `restic_pw_sha256` values and reads neither. **OPEN.** -## 3. R-191 — the off-site tier stops asking to prune +**The capability map DID need touching, and the §N.5 escalation did NOT fire — the row was not +overclaiming.** `00-capability-map.md`'s offsite row already recorded reinstall-continuity: *"a +recreated data volume that orphaned the repo … is now CLASSIFIED … + a move-aside (never-delete) +reset"*, shipped controller v0.142.0 and live-fired 2026-07-18. **That independently corroborates the +Q3 prediction from a different direction.** What it did not record — and now does — is that this fires +on an **ordinary, planned, unattended guest rebuild** on every box, with the measured 2026-08-03/04 +instance, the hash evidence, and the inverted `escrow_stale` wiring. **Status unchanged +(PROVEN-LIVE):** the guard is correct and is not the gap; the cadence is. -**The defect:** `keep_last: 2` on the offsite tier, so vzdump uploaded and then failed the whole job -on `missing Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → -`whole_guest_backup_failed`. R-89 moved offsite pruning server-side; the 2026-07-26 "two weeks" ruling -was not reversed — **where it is enforced moved and this value did not follow.** +`ROADMAP.md`'s newest row is R-171 and none of R-190…R-197 has one, so **there was nothing to +collapse.** One correction *was* warranted under the coupling rule: R-1's row claimed the v0.57.0 +reinstall arc makes offsite re-issue *"auto-fire on re-enrollment"* — scope-corrected in place, since +that means **host** re-enrollment after a host **delete**, and a guest rebuild never reaches it. -- **Installer 1.25.0**: `keep_last: 0`, and the justifying paragraph rewritten to say where retention - now lives, citing R-89 and the Part 0 evidence, plus the warning that if ep0 ever stops pruning this - zero is unbounded growth. The agent's guard `allowPBSPrune = !primary && keep_last > 0` already - reads 0 as *never prune* — **no agent change**. -- **A gate asserts it** (`hostinstall_gates.py`), red-proved: pinning `keep_last: 2` back → - `FAIL: the offsite backup_target arms a CLIENT-SIDE prune (keep_last=2)`. -- **Both live boxes**, quoted: +`CONTEXT.md` gained **S-26** (the spike's Q1/Q2 headlines + the three corrections that must not be +re-inherited) and **S-27** (the R-195 ruling and its mechanism). `REUSE.md` gained the +`HasEverBoundHost` row with its "never a substitute for a liveness check" trap. `STATUS.md` rewritten +for the operator, one screen. -``` -demo-felhom backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true -demo-felhom backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false primary=false -felhom-host backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true -felhom-host backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false primary=false -``` +--- -The local tier is untouched (Scenario C). +## 8. CI, task ids, and `--no-verify` -**Not yet observed:** a weekly offsite run completing OK end to end. The failing step is removed and -the boxes are armed correctly; the *schedule* proving it is next week's event. The row says so rather -than implying the run has happened. +| commit | what | CI task id | run # | conclusion | +|---|---|---|---|---| +| `7fff45d688fa` | R-195 code + tests + the spike + all docs | **126** | 72 | **success** | +| `f581ac134964` | `manifests/hub.yaml` 0.91.1 → 0.92.0 | **127** | 73 | **success** | -## 4. R-190 — the grant repairs itself, and the repair is visible +Matched by `head_sha` via +`curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/felhom.eu/actions/tasks?limit=5"` — a PULL +check, not a reliance on the failure mail. +**`--no-verify` was NOT used.** The pre-push hook ran and reported `gates OK - push proceeding`; +`python3 scripts/repo_gates.py` was additionally run in full (all five gates OK) before committing. -**Mitigation.** On a missing grant the agent runs the existing root wrapper -`felhom-backup-target-apply grant ` and **re-reads once** — the pbsdr R-22 shape and its -restraint. **No new privileged surface:** `grant *` was already in the sudoers allowlist for any -storage id (confirmed in `configs/felhom-agent.sudoers`, not assumed), and the verb already grants -both user and token. It existed, was permitted, and had only ever been called at storage **creation** -— the *built but never wired* shape in a verb rather than a seam. Bounded: one attempt per tier per -hour. +**Teardown:** the spike provisioned nothing — no guest, no VM, no storage, no credential. The only +artefact is a hub-DB snapshot in the session scratchpad, outside the repo. -### The record — and how v0.124.0 got it wrong in production with a green test suite +**Read-only discipline held:** no `ReissueCredentials`, no ceremony, no credential rotation, no +change to any box's offsite configuration, no snapshot deleted, no customer record modified. +`drill-r50` untouched; `sess-f`'s removal **confirmed** rather than assumed. -§8.5 asked whether the hub's existing degraded↔ok edge sufficed **before building anything new**. It -does, as a *channel* — but only if the agent deliberately reports one degraded cycle, because a silent -repair produces no edge at all. So **nothing new was built**: no wire change, no hub change, no new -event type. The explanation rides `Feature`, because that is the field the hub interpolates into the -operator's e-mail (`monitor/host_capability.go` builds its message from names and features; `Reason` -does not travel). +--- -**v0.124.0 implemented "one cycle" as one probe CALL, and that failed live:** +## 9. Observations — noticed and NOT acted on -``` -09:39:34 agent: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED … confirmed_by=re-read -09:39:34 agent: capabilities self-check ok=69 total=70 degraded=1 -09:39:37 hub: host-report stored → pve:store-grant:felhom-backup = ok ← the record never arrived -``` - -`probeAll` is invoked **independently** by the self-check log and by the collector building a report. -The repairing call was the log's; the report three seconds later found the grant present and sent -`ok`. The agent's journal had the record, the hub had nothing, and the operator would have learned -nothing — the exact silence R-190 exists for, re-created inside its own mitigation, **while every unit -test passed**. **v0.124.1** replaces it with a latch on **time** (20 min > the 900 s report interval). - -### Live proof, twice, on demo-felhom - -Grant deleted by hand (both rows — user and token), which reproduces R-190 and the R-185 symptom -(`{"data":[]}` through the token): - -``` -10:00:29 agent: store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190) - target=felhom-backup privilege=Datastore.AllocateSpace - action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read -10:00:30 hub: host-report → pve:store-grant:felhom-backup = DEGRADED - feature: "…the agent's storage grant was MISSING and has been AUTOMATICALLY RESTORED…" -10:00:40 hub: Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded) -10:00:40 hub: Operator email sent for demo-felhom/agent_capability_degraded - pveum acl list | grep -c felhom-backup → 2 (both rows restored) -``` - -**And the pair CLOSED**, which is the other half of "exactly one alert per loss": - -``` -10:20:29 the 20-minute latch expires -10:30:40 hub: Host capability: demo-felhom-8363b5 degraded → ok (agent_capability_recovered) -``` - -So one lost grant produced **one degraded event + one operator e-mail + one recovery**, and nothing -further. Final state on both boxes: agent `0.124.1`, two ACL rows on `/storage/felhom-backup`. - -Scenario E is unchanged from v0.123.0 — a failed repair, or one that does not survive the re-read, -leaves the capability degraded with the failure in its reason. - -## 5. Part 3 — one bounded pass at the mechanism, negatives named - -**The lead is real as a CLASS and is documented in our own installer:** *"`pveum user token remove` -purges the token's ACL, so re-applying post-rotate is mandatory"* — a rotation would take every grant -and re-apply only the hardcoded `PVE_STORAGES` set, which is exactly the observed end state. - -**It does not fit this box.** A rotation purges **all** the token's ACLs and mints a **new secret**; -demo-felhom's token still authenticates with the same secret (`--selftest` OK), it kept its other -three storage grants throughout, and only `felhom-backup` was refused. No installer run is evidenced -(no 2026-08-03 install log; host uptime 12 days at the time). - -**Ruled out, by name:** a host reinstall; any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00; -any cluster-log ACL entry; a token rotation **on this box**. **Not ruled out fleet-wide** — any -installer run still purges and re-grants only `PVE_STORAGES`, though installer 1.24.0's reuse-arm fix -now re-grants the backup target on that path. - -**The mechanism remains open**, and the row says which half it reflects: mitigation shipped, cause -unexplained. No reproduction was attempted and the session was not extended for it. - -## 6. Tests and red-proofs - -Green gate both repos: `go build ./... && go vet ./... && go test ./...` rc=0; `agent_gates.py` and -`repo_gates.py` all OK. Test runs and commits always separate commands. - -| # | Test | Mutation | Observed | -|---|---|---|---| -| G | `hostinstall_gates.py` offsite assertion | pin `keep_last: 2` back | **FAIL** — `arms a CLIENT-SIDE prune (keep_last=2)` | -| F | `TestGrantRepair_IsBounded` | drop the interval check | **FAIL** — `5 cycles produced 5 attempt(s)` | -| D(record) | `TestGrantRepair_ARepairedGrantIsReportedAsATransition` | report `ok` after a repair | **FAIL** — `got "ok" — the loss would be invisible` | -| D(latch) | `TestGrantRepair_TransitionSurvivesALaterProbe` | ignore the latch in the healthy path | **FAIL** — `got "ok" — the host-report would carry ok` | -| H | `TestMainWiresTheGrantRepair` | pass `nil` instead of the repairer | **FAIL** — `not passed the repairer` | -| — | `TestGrantRepair_CallsTheExistingWrapperVerb` | — | pass (argv is exactly `grant `; anything else is a policy change) | -| — | `TestGrantRepair_FailureIsReturned`, `_NilIsSafe` | — | pass | - -**Two red-proofs did not fail on the first attempt, and both are reported rather than quietly fixed:** - -1. **The seam mutation (`grantRepairer` → `nil`) failed to COMPILE** (`declared and not used`), so the - test never ran. The guard holds — you cannot ship it — but it was not demonstrated. The mutation - that does fail it keeps the variable used (`_ = grantRepairer`) and passes `nil`. -2. **The latch red-proof passed**, because the test asserted the latch *helper* rather than the path - consuming it. Fixed by extracting `storeGrantHealthyVerdict` and pointing the test at it — after - which the mutation fails as it should. **This is the second hollow test caught in this file**; the - first asserted a `Status` it had constructed itself. Both are now production functions under test. - -Timestamps in the repair tests carry jitter (`09:39:34`, `+13m07s`, `+27m51s`, …) rather than round -intervals, per §10. - -## 7. Files, commits, tag - -`felhom-agent`: `cmd/felhom-agent/main.go` (repairer, latch, `storeGrantHealthyVerdict`, -`storeGrantRepairedVerdict`, wiring), `cmd/felhom-agent/storegrant_test.go`, `CHANGELOG.md`. -Commits `257c4d8` (v0.124.0) and the v0.124.1 fix. Released shas: **`5e417938…`** (0.124.0), -**`5c279bda…`** (0.124.1), both round-trip verified. - -`felhom.eu`: `scripts/felhom-host-install.sh`, `scripts/hostinstall_gates.py`, `scripts/CHANGELOG.md`, -`manifests/webpage.yaml`, `documentation/backlog/OPEN-ITEMS.md`, -`documentation/architecture/07-backup-architecture.md`, `CONTEXT.md`, `STATUS.md`, `REPORT.md`. -**Installer tag: `installer-v1.25.0`.** - -## 8. The served installer - -``` -$ curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 SCRIPT_VERSION -SCRIPT_VERSION="1.25.0" -$ … | grep -o '"keep_last":[0-9]*' -"keep_last":0 -``` - -Both `--ref=` lines bumped to `installer-v1.25.0` (327, 372); ArgoCD Synced/Healthy; webpage rollout -complete. The served bytes carry the change, not just the version. - -## 9. Deployed versions - -Both demo hosts run **`felhom-agent 0.124.1`**, deployed from the **published** artifact (downloaded -from Gitea; deployed sha matches the published sha on both). `systemctl is-active` → active on both. -**NOT VOUCHED** — that stays the operator's act. - -## 10. Registers - -- **R-191 → CLOSED** (installer 1.25.0 + both boxes), carrying the Part 0 evidence *and* the - broken-instrument warning, because the latter is the more reusable lesson. -- **R-190 → MITIGATION SHIPPED, MECHANISM STILL OPEN** — stated in the status field itself so the row - cannot be misread as solved. -- **R-194 → filed** (grep established it free): PVE caches permissions — ~40 s in one run and - ~16 minutes in another between deleting both ACL rows and the permission read reflecting it. It puts - a floor under detection speed, makes a single permission read a lagging indicator, and is a - candidate explanation for *when* R-190's refusal surfaced (not for what removed the grant). -- `07-backup-architecture.md`: the offsite retention row was **already correct** — the configuration - disagreed with it. Annotated, because a doc that states a contract does not enforce it; the gate does. -- `ROADMAP.md` holds none of these rows. - -## 11. Teardown - -The hand-removed grant was restored **by the agent**, which is the test, and confirmed present (2 ACL -rows). Nothing else was provisioned: no scratch storage, no fixture, no prune job, no probe tag. -Nothing was deleted from ep0 or from any tier. - -## 12. Observations — noticed, NOT acted on - -- **PVE's permission cache is the reason detection is not instant** (R-194). Notably, the storage - *content* listing reflected the loss immediately (`{"data":[]}`) while the *permission* read still - said present — two signals disagreeing, and today only one is read. Worth deciding whether the probe - should consult both. -- **`logCapabilities` runs `probeAll` independently of the report path.** The latch makes that safe - for this capability, but any future probe with one-shot semantics will hit the same trap. The - general fix would be one probe per report cycle, shared — deliberately not attempted here. -- **demo-hp's offsite tier is still `escrow_state: escrowed, snapshot_count: 0`** after this morning's - re-issue; whether the pre-rebuild 15 snapshots reattach is answered by its next offbox run (04:15), - and belongs to R-193 rather than to this session. -- **The `felhom-pbs` tier is armed on both boxes with `cadence=168h`**, so the R-191 fix cannot be - observed end-to-end until that cadence fires. Nothing was triggered manually to shortcut it. +1. **The orphaned ciphertext is retained and keeps consuming quota, with nothing to prune it.** + `resetOrphanedRepo` is move-aside-never-delete by design (correct), but 40.9 MB + 1.14 GB of + unopenable data will sit against a 50 GB soft quota indefinitely. S-24's ep0 prune jobs cover the + **PBS** namespaces, a different tier. Not filed — it belongs to whichever option is chosen for + R-193. +2. **`host_escrow_superseded` holds only two rows in the entire database**, both from 2026-08-04. That + is why the spike could not establish whether demo-felhom's 2026-07-21 Re-issue re-sealed an + *unchanged* password — the predicted false-staleness shape of R-196. Retention policy for that + table was not investigated. +3. **Both demo guests were rebuilt on consecutive mornings** (demo-hp 08-03 06:12, demo-felhom 08-03 + 07:17, both to controller 0.192.0 with new config hashes). *Why* is outside this spike's scope and + was not chased. +4. **`api/handler.go:1067-1069`'s comment is the third copy of R-196's false premise**, and it sits in + a different package from the other two. Whoever fixes R-196 should grep rather than fix the one + they were shown. +5. **`peti-felhom` is `active` with its host deleted since 2026-07-15**, judged only by the `down` + skip. It is correct today and would start alarming the moment it reported once. Not a defect, but + the shape is worth knowing. +6. **The `-shm`/`-wal` trap generalises.** Any future session reading the hub DB from the pod must + copy all three files. Reading `hub.db` alone today would have produced confident answers from a + file 2 h 15 m stale — the R-3 shape with a different filename. diff --git a/STATUS.md b/STATUS.md index 994d7ce..11522f3 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-08-03.** +**Updated 2026-08-04.** > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this > page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**, @@ -20,10 +20,21 @@ Proven end to end on real hardware. ## What's broken -- **Rebuilding a machine silently takes away its off-site app-data backup.** Fixed on demo-hp on - 4 August (re-issued, picked up in 15 seconds, same repository, you escrowed the key), but the - underlying fault stands: the password is delivered once and a rebuilt machine cannot ask again. - The daily email about it also tells the wrong story. *(R-193, R-192)* +- **Rebuilding a machine throws away its off-site backup HISTORY — and that is worse than we thought + yesterday.** We spent today establishing what a rebuild actually destroys, and the answer changes the + picture. There are three secrets involved. The one we have been talking about — the password to the + storage provider — is the one you can re-issue any time, and we did. **The one that matters is the + key that encrypts the backups themselves. The machine invents that key, and a rebuilt machine + invents a brand-new one**, so the old backups can no longer be opened by the machine that wrote + them. + **Both demo machines did exactly this, and one of them did it silently.** demo-hp lost access to + 15 backups (41 MB). demo-felhom — the one we called lucky, because its off-site copy came back on + its own in 76 seconds — **lost access to 36 backups (1.14 GB), and nothing said a word for + thirteen hours.** The luck restored the delivery, not the history. + Nothing has been deleted, and this was measured without touching either machine or any password. + **The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start + a new history** — the machine already knows how to recognise this and asks before resetting. + **There is a decision here for you — see "Waiting on you".** *(R-193, R-192, and two new items)* - *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing @@ -34,6 +45,14 @@ Proven end to end on real hardware. ## What shipped recently +- **The daily error email about David is gone.** David's record was created when he said he would try + Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every + morning at five the system reported that a database backup had not happened. **A customer with no + machine ever attached now produces no alarm.** A machine that *was* attached and went quiet still + does, because that is a real fault, and there is a test whose whole job is to make sure a future + change cannot take that away. David's record was not touched — the record was right; the alarm was + wrong. **The first quiet morning will be 5 August, so this is not confirmed yet.** *(R-195)* + - **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we found a permission that vanished with no reinstall and no recorded action; the machine now notices, restores it through the same controlled helper that grants it in the first place, and **raises an @@ -58,7 +77,12 @@ Proven end to end on real hardware. ## What we're working on -- **Now:** nothing outstanding. +- **Now:** waiting on your answer about the backup key (above). Two small honesty fixes are ready to + go the moment you say so — making a changed backup key raise an alarm, and correcting an email that + currently states the opposite of what it measured. Neither needs a decision. +- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to + refuse and ask, rather than start fresh — but that is a prediction from the code, not something + anyone has watched happen. - **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing tests unattended. Most of the machinery it needed arrived with the restore-test change below. *(R-87)* @@ -66,6 +90,22 @@ Proven end to end on real hardware. ## Waiting on you +- **A real decision, and it is the important one on this page.** Every machine is going to be + reinstalled at some point, and each reinstall throws away the key that opens its off-site backups. + There are only two honest answers, and picking either is fine — picking neither is what leaves it + where it is. + **(a) Keep a copy of that key on the Proxmox host**, which survives a rebuild. The machinery for + this already exists and is barely any work. The price: the key would then sit on a machine you have + root on, so "only the customer's recovery code can open these backups" stops being true in the + strict sense. It was never quite true anyway — you hold root everywhere — but this makes a real + second copy, and copies are exactly what we are already uneasy about elsewhere. + **(b) Leave the key sealed under the customer's recovery code**, accept that every rebuild starts a + fresh off-site history, and make sure the system says so loudly on the day it happens instead of + never. Old backups are kept, not deleted, but they keep consuming the customer's off-site space. + **We recommend doing the "says so loudly" part immediately regardless** — it is cheap, it costs + nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen + hours. What we are asking you to choose is (a). *(R-193; the full reasoning is in the findings + document)* - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a session log; nothing suggests anyone else saw it. *(R-132)* - **One small question, not urgent.** The automatic check cannot see which version you have told @@ -75,6 +115,10 @@ Proven end to end on real hardware. ## Changed since last update +- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage + password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost + access to their off-site history, one of them silently. A decision is now waiting on you. The daily + false alarm about David is fixed. *(R-193, R-195)* - **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own, overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the weekly off-site backup reports failure after a successful upload. *(R-185, R-191)* diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 7f26359..0311672 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -26,7 +26,7 @@ | R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | | R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | | R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 | -| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path | +| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path. **SCOPE-CORRECTED 2026-08-04 (R-193 spike, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`): "auto-fire on re-enrollment" means HOST re-enrollment, and only when the host record was DELETED first.** `reissueOnReenroll` sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`, `api/handler.go:1004-1016`), so a **guest** rebuild — which leaves the `hosts` row intact — never reaches F2 or F3. Measured live: demo-hp's guest was rebuilt 2026-08-03, `demo-hp-bb76ea` was unchanged throughout, F3 did not fire, and the box had **no off-site copy for 25 hours** until an operator ran Re-issue by hand. Peti's clean-slate (a real host delete + re-enroll) still walks the supported path; **the arc does not cover a guest rebuild, and that is the gap R-193 owns.** Note also that F3 alone would NOT have saved the data — the rebuilt controller mints a fresh restic repo password regardless (spike Q2) | | R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) | | R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b | | R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |