R-195: a customer with no machine ever bound does not alarm (hub v0.92.0) + R-193/R-192 spike
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0, reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings running. The existing down-skip could never cover it: it reads the staleness checker's state, which is seeded from a query over the `reports` table, so a customer that never reported has no state at all and GetState() returns "" rather than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is consulted once per customer at the top of the deadline loop. The discriminator is "was a host EVER bound", never "has a report arrived" — a box installed and never heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the event david sent. Parts 0-3 (spike, NO production code for R-193/R-192): audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot provider password is the RECOVERABLE secret and the restic repository password is the irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous off-site history. Measured without touching a box, by comparing host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed (demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's "lucky" 76-second recovery restored delivery and not the repository, silently, for 13h. ReissueCredentials does NOT rotate the restic password (R-39's record and two hub comments are wrong -> R-196); candidate (b) is not implementable against a zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong event. Ends in ranked options and an unanswered question for the operator. R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed.
This commit is contained in:
+58
@@ -17,6 +17,64 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-27 — a customer with NO machine ever bound is UNKNOWN, silently; one that was bound and went quiet
|
||||
still alarms (2026-08-04, R-195; hub v0.92.0).** Operator ruling, implemented as
|
||||
`store.HasEverBoundHost` (live `hosts` row OR `host_deletions` tombstone) consulted once at the top of
|
||||
`CheckBackupDeadlines`' loop. **The discriminator is "was a host EVER bound", never "has a report
|
||||
arrived"** — a box that was installed and never phoned home is a real fault. Fail-**open** on a read
|
||||
error; the deferral is logged with its own counter; the R-81 anchored-verdict structure is untouched.
|
||||
|
||||
*The mechanism is worth carrying, because the guard that should have covered this was keyed off the
|
||||
wrong thing:* the existing down-skip reads `StalenessChecker.GetState()`, whose map is seeded from
|
||||
`store.GetCustomers()` — **a query over the `reports` table**. A customer with zero reports is in no
|
||||
row, gets no state, and `GetState()` returns `""` not `"down"`, so **the skip misses exactly the
|
||||
customer it would most obviously cover.** `david` (created 2026-08-01, no machine) e-mailed an
|
||||
`expected_dbdump_missed` ERROR three mornings running; `peti-felhom` — active, host deleted
|
||||
2026-07-15 — does not, because its 482 old reports make it `down`. Generalise it: **a "skip the dead"
|
||||
guard built on evidence of life cannot see something that was never alive.**
|
||||
|
||||
**S-26 — the one-shot secret is the recoverable one; the irreplaceable one is minted fresh on every
|
||||
guest rebuild (2026-08-04, R-193 spike — `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`).
|
||||
No code shipped for it; the decision is the operator's.**
|
||||
|
||||
Two sentences, because they are what every future session needs before touching this area:
|
||||
|
||||
- **Q1 — what is one-shot.** Of the three secrets in the offsite path, the hub's **one-time
|
||||
provider password** is one-shot but freely re-issuable, the box's **SFTP key** is regenerated on
|
||||
every apply by design, and the **restic repository password** — the DATA key, which the agent's own
|
||||
source calls *"irreplaceable"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the only one
|
||||
nothing can restage.
|
||||
- **Q2 — what a rebuild costs.** A rebuilt controller **mints a brand-new repository password**
|
||||
(`WriteOffboxSecrets`, `offbox.go:392`, generates whenever `<DataDir>/offbox/repo_password` is
|
||||
absent) and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly
|
||||
one caller in the whole repo, a web form a human pastes into — so **every guest rebuild orphans the
|
||||
previous off-site history.**
|
||||
|
||||
*Measured without touching a box, and the method is reusable:* `host_escrow.restic_pw_sha256` vs
|
||||
`host_escrow_superseded.restic_pw_sha256`. demo-hp `8e03eddf…`→`8a9e33aa…` (15 snapshots / 40.9 MB
|
||||
orphaned); demo-felhom `48741892…`→`c60c8bc7…` (**36 snapshots / 1.14 GB**). **demo-felhom is the half
|
||||
that matters:** R-193 recorded it as having survived the rebuild by luck, and it did — its *delivery*
|
||||
recovered in 76 s off a stale staged secret. **Its repository did not**, and nothing said so for 13 h.
|
||||
*Luck restored the plumbing, not the data.*
|
||||
|
||||
*Three corrections that must not be re-inherited.* **(a)** `ReissueCredentials` does **not** rotate the
|
||||
restic password — R-39's record, `offsite.go:198-201` and `api/handler.go:1067-1069` all say it does,
|
||||
and all three are wrong (→ **R-196**, the eighth entry in `CLAUDE.md`'s table). **(b)** Candidate (b),
|
||||
*"recoverable from escrow at re-bootstrap"*, is **not implementable** — the escrow is R-wrapped and
|
||||
zero-knowledge (D6), so only a customer-present ceremony can open it, which is the manual form that
|
||||
already exists. **(c)** Candidate (a), *"the hub restages automatically"*, **already exists** as
|
||||
`reissueOnReenroll`'s F3 leg and is wired to the wrong event — it sits behind `handleHostEnroll`'s
|
||||
mint-once-reuse short-circuit, and a **guest** rebuild leaves the `hosts` row intact. **Shipping it
|
||||
first would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** —
|
||||
strictly worse than the current loud failure.
|
||||
|
||||
*The candidate nobody had named,* and the only one aimed at the actual harm: **the agent survives a
|
||||
guest rebuild**, already receives the repo password over the pinned local API
|
||||
(`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after
|
||||
the ceremony. Retaining and serving it back needs no new seam. Its price is one real trade: a copy of
|
||||
the data key at rest on the Proxmox host. **That trade is the operator's to make and the spike does not
|
||||
make it.**
|
||||
|
||||
**S-24 — offsite retention is ep0's, and the box asks for none (2026-08-04, R-191; installer 1.25.0).**
|
||||
R-89 moved offsite pruning server-side and box tokens stay write-only. The 2026-07-26 "two weeks"
|
||||
ruling was not reversed — **where it is ENFORCED moved, and the installer's `keep_last: 2` did not
|
||||
|
||||
@@ -75,6 +75,7 @@
|
||||
| `(*Store).GetHostRecoveryMeta` + `(*Server).handleHostRevealRecoveryCredential` | hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | `(hostID) (*HostRecoveryMeta, error)` · `POST /hosts/{id}/reveal-recovery-credential` | The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | **Use `GetHostRecoveryMeta` on any page-render path** — its struct and its `SELECT` both omit the `secret` column, so it cannot leak one; `GetHostRecoveryCredential` (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE `recovery_credential_revealed` event via `SaveEvent` and calls NO dispatcher (the `handleRequestLogTail` shape). `api/handler.go handleAdminGetRecoveryCredential` (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
|
||||
| `host_detail_body` sub-template | hub/internal/web/templates/host_detail_body.html | `{{template "host_detail_body" .}}` | Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with `{{.HostID}}` (the customer page renders N instances). |
|
||||
| `(*Store).ListHostsByCustomer` | hub/internal/store/store.go (~L1620) | `(customerID) ([]Host, error)` | A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
|
||||
| `(*Store).HasEverBoundHost` (v0.92.0, R-195) | hub/internal/store/store.go | `(customerID) (bool, error)` | Any verdict that must not fire for a customer with **no machine ever bound** — "was anything ever expected of this customer" | `hosts` row **OR** `host_deletions` tombstone. **NOT a liveness check and never a substitute for one:** a box that was bound and went silent returns `true` and must keep alarming — that is the case any change here breaks first (pinned by `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`). Callers **fail OPEN** on its error: an unreadable binding must never SUPPRESS an alarm. Do **not** re-derive this from report presence — `store.GetCustomers()` (and therefore the staleness checker's `down` state) is a query over `reports`, so a never-reported customer has no state at all, which is exactly how the daily false alarm reached `david`. |
|
||||
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
||||
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
||||
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
||||
|
||||
@@ -76,7 +76,7 @@
|
||||
| Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded **DEGRADED** rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | `E2D-fresh-vm-2026-07-29` C1 (real 1.22.0 install, rc=0, `Day-0 provision SUCCESS`) + C2 (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort) | Case A (a second drive already present at install) has never fired naturally — only Case B has |
|
||||
| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 **DB replay route → `07-backup-architecture.md` §8 row 3** |
|
||||
| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | **Route + RTO → `07-backup-architecture.md` §8 rows 1, 2, 4, 5.** The matrix records that the copy's `recovery-unit/` mirror is read by no path (→ R-102) |
|
||||
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect) |
|
||||
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect). **2026-08-04 (R-193/R-197, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`) — the row is NOT overclaiming and the guard is not the gap; the CADENCE is.** This row already recorded that a recreated data volume orphans the repo, and the spike confirms the mechanism at source: `WriteOffboxSecrets` (`offbox.go:392`) mints a fresh 256-bit repo password whenever `<DataDir>/offbox/repo_password` is absent, and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly one caller, a web form a human pastes into. What was NOT recorded is that this now fires on an **ordinary, planned, unattended guest rebuild**, on every box: measured on BOTH demo boxes 2026-08-03/04 by comparing `host_escrow.restic_pw_sha256` against `host_escrow_superseded.restic_pw_sha256` (demo-hp `8e03eddf…`→`8a9e33aa…`, demo-felhom `48741892…`→`c60c8bc7…`), orphaning **15 snapshots / 40.9 MB** and **36 snapshots / 1.14 GB** respectively. **demo-felhom is the important half:** it kept its DELIVERY (a stale staged secret restored the target in 76 s) and lost its REPOSITORY anyway, with **nothing marking the escrow stale for 13 h** — `escrow_stale` is wired to `ReissueCredentials`, the one path that does NOT change the repo password (**R-196**), and absent from the rebuild path that does. Status unchanged: the classify-and-move-aside guard remains PROVEN-LIVE and correct, and is predicted (not yet measured) to refuse the 2026-08-05 run on both boxes rather than start a silent fresh history |
|
||||
| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** (2026-07-20) | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* **2026-07-20: the clean run HAPPENED** — endpoint-level supervised reconstitute of immich from snapshot `49e7cb46` (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no `already exists`, operation reported SUCCESS, immich's own DatabaseService logged `No schema drift detected` twice, 11 assets `active`, 4/4 containers healthy, 231 `public` indexes. **Operator confirmed the immich timeline renders correctly after the reconstitute** (screenshot held, 2026-07-20). Evidence: `felhom-controller/REPORT.md` §4b. **2026-07-20, LATER THE SAME DAY — the destructive drill RAN and the row now earns PROVEN-LIVE.** The operator deleted the photos in immich own UI **and emptied the trash** (the step whose absence makes a drill prove nothing — the round-1 lesson), then restored through the customer-facing UI. **`40 file(s) placed`** against the 6 of the earlier non-destructive run — the files were really gone and really came back — plus 1 DB dump replayed rc-0, 11 assets `active`, `No schema drift detected`, timeline confirmed by the operator. This is the destroy-then-recover proof the 6D downgrade asked for, and it was taken through the customer own buttons, not endpoint shortcuts. Evidence: `felhom-controller/REPORT.md` 4e **Route + RTO → `07-backup-architecture.md` §8 rows 3, 4** — the matrix also records that no offsite action unpacks the named-volume tars it captures (→ R-107) |
|
||||
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
|
||||
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill **Route + RTO → `07-backup-architecture.md` §8 rows 6, 8, 9** — measured 84–112 s local / 1101 s PBS into a scratch guest; a restore to a DIFFERENT host is unmeasured |
|
||||
|
||||
@@ -0,0 +1,501 @@
|
||||
# SPIKE — what is actually one-shot, and what a guest rebuild really costs
|
||||
|
||||
**Date:** 2026-08-04 · **Items:** R-193 (rebuild drops offsite), R-192 (the alert says the opposite of
|
||||
what it measured) · **Class:** spike — **no production code shipped for R-193 or R-192**
|
||||
**Baselines read:** `felhom.eu` @ `f456835bbcc4` (hub v0.91.1) · `felhom-controller` @ `0887fd67` ·
|
||||
`felhom-agent` @ `856a127`
|
||||
|
||||
> **This document ends in ranked options and a STOP.** It establishes facts; it does not choose.
|
||||
> The operator's question is stated, unanswered, at the end.
|
||||
|
||||
---
|
||||
|
||||
## 0. Why this exists
|
||||
|
||||
Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
|
||||
session reports described the same mechanism in contradictory terms:
|
||||
|
||||
- R-39's record: Re-issue *"rotates the restic password and makes the escrow stale"*.
|
||||
- R-193's own resolution note: the one-time password is *"only the transport credential used once to
|
||||
install the box's own SSH key"*.
|
||||
|
||||
Those two worlds imply very different fates for the fifteen existing snapshots. This spike settles it
|
||||
**from source first, live state second**, so a measurement can contradict the code rather than be
|
||||
interpreted by it.
|
||||
|
||||
**The headline, before the detail:** *both prior claims are wrong in the way that matters.* Re-issue
|
||||
does **not** touch the restic repository password — but a **guest rebuild does**, silently, on every
|
||||
box, with nothing marking the escrow stale and nothing telling anyone. Both demo boxes lost repository
|
||||
continuity on 2026-08-03/04. The one that "recovered by luck" lost it too.
|
||||
|
||||
---
|
||||
|
||||
## 1. The instruments, and what each one can and cannot answer
|
||||
|
||||
Stated up front so a wrong instrument is visible here rather than inside a conclusion (§8 rule 2).
|
||||
|
||||
| # | Instrument | Exact query / read | Known limit |
|
||||
|---|---|---|---|
|
||||
| I1 | hub SQLite snapshot | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale at copy time (mtime 08:34 vs wal 10:45 CEST). Copying only the main file would have produced confident answers from a stale file — the exact R-3 shape. |
|
||||
| I1-check | freshness proof | `PRAGMA integrity_check` → `ok`; `MAX(host_reports.received_at)` = `2026-08-04 08:45:38`, `datetime('now')` = `08:49:46` | **Positive observable**: newest row 4 min old. Not "the query returned no error". |
|
||||
| I2 | `host_escrow.restic_pw_sha256` + `host_escrow_superseded.restic_pw_sha256` | see §2 | sha256 of a 256-bit random secret — non-reversible, safe to compare. Answers *"did the repo password change"* **without touching a box or a credential**. |
|
||||
| I3 | `reports.report_json → $.offsite` | grouped distinct-object query with first/last seen | The report echoes the box's own target state; absence of the key = no target configured. |
|
||||
| I4 | `one_time_secrets` | `SELECT customer_id, created_at, consumed_at` | **`customer_id` is the PRIMARY KEY** — one row per customer, last-write-wins. It **cannot** answer "how many credentials were delivered"; a Re-issue destroys the prior row. demo-hp's 2026-07-23 consume is no longer in this table. |
|
||||
| I5 | source | `hub/internal/offsite/*`, `hub/internal/api/*`, `controller/internal/{backup,offsiteapply}`, `agent/internal/escrow` | — |
|
||||
|
||||
**Not used, deliberately:** no `ReissueCredentials`, no ceremony, no config change, no deletion, no
|
||||
write to any box. Nothing on either box or on the storage endpoint was altered by this session.
|
||||
|
||||
---
|
||||
|
||||
## Q2 — Does a rebuilt controller reuse the escrowed repository password, or mint a new one?
|
||||
|
||||
### **RULING: it MINTS A NEW ONE. The escrowed password is never consulted on any automatic path.**
|
||||
|
||||
**Method — source.** The apply-bridge's terminal step is
|
||||
`Enabler.ConfigureOffbox(...)` (`controller/internal/offsiteapply/offsiteapply.go:221,256`), which
|
||||
reaches `Manager.ApplyOffsiteTarget` → `WriteOffboxSecrets`
|
||||
(`controller/internal/backup/offbox.go:370`). Quoted verbatim, `offbox.go:392-401`:
|
||||
|
||||
```go
|
||||
// Auto-generate the repo password once (0600), never log it.
|
||||
if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) {
|
||||
pw, gerr := generateOffboxPassword()
|
||||
...
|
||||
if werr := os.WriteFile(m.offboxPwPath(), []byte(pw), 0o600); werr != nil {
|
||||
```
|
||||
|
||||
`offboxPwPath()` is `<DataDir>/offbox/repo_password` (`offbox.go:363`). A rebuilt guest has a fresh
|
||||
data dir, so the file is absent, so **a fresh 256-bit password is minted**.
|
||||
|
||||
**The only path that recovers the old one is `InjectOffboxPassword`** (`offbox.go:541`), whose sole
|
||||
caller in the entire repo is `offboxInjectPasswordHandler`
|
||||
(`controller/internal/web/offbox_handlers.go:189`) — a **web form where a human pastes the recovered
|
||||
password**. Verified by census: `grep -rn "InjectOffboxPassword" --include=*.go .` → three hits, one
|
||||
definition, one caller, one comment. **The apply-bridge never calls it.**
|
||||
|
||||
**Method — live, and it is decisive without touching a box (I2).** The hub already stores the sha256
|
||||
of the escrowed repo password, so the question is answerable as a hash comparison:
|
||||
|
||||
| host | superseded (pre-rebuild) | current (post-rebuild) | verdict |
|
||||
|---|---|---|---|
|
||||
| `demo-hp-bb76ea` | `8e03eddf9ff7…` created `2026-07-23T10:01:17Z` | `8a9e33aa4da6…` created `2026-08-04T07:15:36Z` | **CHANGED** |
|
||||
| `demo-felhom-8363b5` | `48741892f0ef…` created `2026-07-21T08:38:33Z` | `c60c8bc737a6…` created `2026-08-04T07:20:07Z` | **CHANGED** |
|
||||
|
||||
**Both boxes minted a new repository password.** demo-hp's pre-rebuild escrow (`8e03…`, sealed eight
|
||||
minutes after its 2026-07-23 09:53:41 apply) is the key to its 15 snapshots. That key now exists
|
||||
**only inside a superseded, R-wrapped escrow blob** that the hub cannot open.
|
||||
|
||||
### **This is bigger than R-193 as filed, and it hits the box R-193 called lucky.**
|
||||
|
||||
R-193 records demo-felhom as having *"survived the SAME rebuild by luck"* — 76 seconds of downtime.
|
||||
Measured (I3), demo-felhom's pre-rebuild offsite object was:
|
||||
|
||||
```
|
||||
last_run 2026-08-03T02:17:44Z · last_status ok · snapshot_count 36 · repo_size_bytes 1 136 685 919
|
||||
```
|
||||
|
||||
**36 snapshots, 1.14 GB.** Since 2026-08-03 07:19:10 UTC it has reported `snapshot_count: 0,
|
||||
repo_size_bytes: 0` in every one of 109 reports, and its repo password hash changed.
|
||||
|
||||
> **The luck recovered DELIVERY. It did not recover the REPOSITORY.** The staged secret restored the
|
||||
> transport in 76 seconds and the box then minted a brand-new repo password anyway. The contrast
|
||||
> R-193 draws between the two boxes is real for the *credential*, and **false for the data**: both
|
||||
> boxes lost repository continuity, one loudly and one silently. The silent one is worse.
|
||||
|
||||
---
|
||||
|
||||
## Q3 — What did the 04:15 run report?
|
||||
|
||||
### **RULING: UNMEASURED — the decisive run has not happened yet. And the binary the question offers is the wrong one; source says the answer will be NEITHER 15 nor 1.**
|
||||
|
||||
**Why it has not happened.** The scheduled off-box run fires at ~02:15 UTC (04:15 CEST). Measured
|
||||
from I3 — `last_run` values `2026-08-02T02:15:52Z`, `2026-08-03T02:16:39Z` on demo-hp;
|
||||
`2026-08-01T02:17:54Z`, `2026-08-02T02:17:01Z`, `2026-08-03T02:17:44Z` on demo-felhom. **Daily, both
|
||||
boxes.** Then:
|
||||
|
||||
| box | why 2026-08-04 02:15 UTC produced nothing |
|
||||
|---|---|
|
||||
| demo-hp | no offsite target existed at all — the `offsite` key was **ABSENT** from every report between `2026-08-03 06:12:19` and `2026-08-04 07:12:01` UTC (111 reports). The re-issue landed at 07:15:47, five hours after the window. |
|
||||
| demo-felhom | the target existed but `escrow_state: pending` from `2026-08-03 07:19:10` to `2026-08-04 07:20:13`. `OffboxRunnable` requires configured **AND** escrowed (`offbox.go:564-569`), so the fork-4 gate blocked the run. The ceremony landed at 07:20:28. |
|
||||
|
||||
Both boxes became runnable only this morning. **The decisive run is 2026-08-05 ~02:15 UTC on both.**
|
||||
Reported as unmeasured rather than inferred.
|
||||
|
||||
### What source says will happen — and it is a third outcome
|
||||
|
||||
Same sub-account, same repo path, new password. Measured coordinates (I3, `$.dr_recipe`):
|
||||
`u629488-sub3.your-storagebox.de:/home/felhom-repo` — **unchanged** across the whole incident, and
|
||||
`repoPath` is a compile-time constant (`hub/internal/offsite/offsite.go:85`). So the next run opens an
|
||||
**existing repo with the wrong key**, which the codebase already has a name for
|
||||
(`controller/internal/backup/offbox.go:67-93`):
|
||||
|
||||
```go
|
||||
// ErrOffboxOrphaned is the sentinel returned when the offsite repo exists but is keyed under a
|
||||
// passphrase this controller no longer has (the reinstall shape) ...
|
||||
case strings.Contains(s, "wrong password or no key found"):
|
||||
return "orphaned"
|
||||
```
|
||||
|
||||
and `ensureOffboxRepo` (`offbox.go:663-679`) branches on claim state:
|
||||
|
||||
```go
|
||||
if !m.settings.GetClaimed() { // UNCLAIMED → auto move-aside + re-init
|
||||
...
|
||||
m.markOrphaned()
|
||||
return ErrOffboxOrphaned // CLAIMED → skip the run, show the orphan card
|
||||
```
|
||||
|
||||
Both boxes report `claimed: 1` (I3, `$.claimed`, latest report each). **So the predicted outcome is:
|
||||
the run REFUSES with `ErrOffboxOrphaned`, produces no snapshot at all, and waits for the customer to
|
||||
confirm a reset.** Not 15 (reattached) and not 1 (silently fresh) — a **third** outcome the question's
|
||||
framing did not contain. That is the good news half: the system is not going to quietly start a new
|
||||
history over the old one. It will stop and say so.
|
||||
|
||||
**Consequence if the customer confirms the reset:** `resetOrphanedRepo` (`offbox.go:270-314`) moves the
|
||||
old repo aside and re-inits — *"move-aside, not deleted"*. The 40.9 MB + 1.14 GB of old ciphertext
|
||||
survives, unreadable without the superseded escrow, **and keeps consuming the 50 GB soft quota
|
||||
indefinitely.** Nothing prunes a moved-aside restic repo (S-24 covers ep0's PBS namespaces, a
|
||||
different tier).
|
||||
|
||||
**To measure it tomorrow:** re-read `$.offsite.last_status` / `last_error` on both boxes after
|
||||
02:20 UTC, and `$.offsite.repo_state` for the orphan flag. **Predicted `ErrOffboxOrphaned`, not a
|
||||
snapshot count.** Whoever picks this up should record which of the three actually occurred — a
|
||||
prediction from source is not a measurement.
|
||||
|
||||
---
|
||||
|
||||
## Q1 — What is one-shot, and what is not?
|
||||
|
||||
Three secrets, and they have almost nothing in common.
|
||||
|
||||
| | **① storage-provider password** | **② the box's SFTP key** | **③ restic repository password** |
|
||||
|---|---|---|---|
|
||||
| **What it is** | the Hetzner sub-account / box password | ed25519 keypair for `sftp` transport | the **data** key — restic's repo encryption |
|
||||
| **Generated by** | **hub** — `genPassword()` (`offsite.go:481`) | **controller** — `KeyGen.Generate()` (`offsiteapply.go:234`) | **controller** — `generateOffboxPassword()`, 32 random bytes hex (`offbox.go:406`) |
|
||||
| **Stored where** | `one_time_secrets` (hub, plaintext, **one row per customer**) | `<DataDir>/offbox/ssh_key` 0600 (box only) | `<DataDir>/offbox/repo_password` 0600 (box) + inside the R-wrapped escrow blob (hub, opaque) |
|
||||
| **Delivered how** | `GET …/offsite/consume-password`, **served exactly once** then marked consumed (`api/offsite.go:9-31`) | never delivered — installed onto the provider by ssh-copy-id using ① | never delivered anywhere; it never leaves the box except into the escrow ceremony |
|
||||
| **Re-issuable?** | **YES** — `ReissueCredentials` resets it at the provider, any time, operator-initiated | **YES** — regenerated on every full apply; `dr_recipe.go:45` states it plainly: *"the SFTP access key is regenerated at DR"* | **NO automatic path.** Recoverable only by unsealing the escrow with the customer's recovery code and pasting it into `offboxInjectPasswordHandler` |
|
||||
| **What a guest rebuild does to it** | nothing (it lives on the hub / at the provider) | destroys it; a fresh one is minted and installed — **harmless** | **destroys it; a fresh one is minted — CATASTROPHIC for the existing repo** |
|
||||
| **Escrowed?** | no | no | **yes** — `IdentityBundle.ResticRepoPassword` |
|
||||
|
||||
The agent's own source names the asymmetry, and it is the sentence the whole spike turns on
|
||||
(`felhom-agent/internal/escrow/identity.go:35-39`):
|
||||
|
||||
> `ResticRepoPassword` … **It is the DATA key for the offsite tier — irreplaceable (unlike the SFTP
|
||||
> access key, which is regenerable at DR).**
|
||||
|
||||
**The one-shot thing (①) is the recoverable one. The irreplaceable thing (③) is the one nothing
|
||||
re-stages.** R-193 was filed against ①. The damage is in ③.
|
||||
|
||||
**Instrument caveat, worth carrying (I4):** `one_time_secrets` has `customer_id` as PRIMARY KEY, and
|
||||
`SaveOneTimeSecret` is last-write-wins **by design** (the R-39(a) guard comment in
|
||||
`monitor/offsite_delivery.go:26-30` depends on it). So the delivery ledger holds only the newest
|
||||
credential per customer: demo-hp's 2026-07-23 consume, on which R-192's whole diagnosis rests, was
|
||||
**overwritten** by the 2026-08-04 Re-issue and is no longer readable from that table.
|
||||
|
||||
---
|
||||
|
||||
## Q4 — Which claim about Re-issue is true?
|
||||
|
||||
### **RULING: `ReissueCredentials` touches ONLY the provider credential (①). It does NOT touch the restic repository password. R-39's record is wrong — and the hub's own comments repeat the same wrong claim in three places.**
|
||||
|
||||
**Method — source, quoted rather than summarised.** `ReissueCredentials`
|
||||
(`hub/internal/offsite/offsite.go:150-228`) does exactly four things:
|
||||
|
||||
1. `genPassword()`;
|
||||
2. `ResetSubaccountPassword` / `ResetBoxPassword` + `WaitAction` — **the provider account password**;
|
||||
3. `Store.SaveOneTimeSecret(customerID, pw)` — stage ① for delivery;
|
||||
4. bookkeeping: `MarkEscrowStale` + `offsite_reissued` + `escrow_stale` events.
|
||||
|
||||
There is **no reference to a restic password anywhere in the function**, and none is possible — the
|
||||
repo password is generated on the box and never leaves it except into the escrow blob. The hub does
|
||||
not hold it in any openable form (`api/handler.go`: *"The hub stores the bytes and NEVER decrypts them
|
||||
(it has no recovery code)"*).
|
||||
|
||||
**But step 4 is justified by a claim that is false**, `offsite.go:198-201`:
|
||||
|
||||
> `// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing`
|
||||
> `// key-escrow blob — which sealed the OLD password — is now STALE.`
|
||||
|
||||
The escrow *is* marked stale; the stated reason for marking it is not something this function does.
|
||||
The same false premise appears twice more: `api/handler.go:1067-1069` (*"The re-issuer resets the
|
||||
restic repo password, which makes the OLD escrow blob stale"*) and `OPEN-ITEMS.md`'s R-193 row, which
|
||||
inherited it from R-39.
|
||||
|
||||
> **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does
|
||||
> not provide** — and the first where the comment is not merely unenforced but factually describes a
|
||||
> different function. It survived because it reads as settled and because its *effect* (a stale escrow)
|
||||
> is real, so nobody checked the *cause*.
|
||||
|
||||
**Consequence, and it is a live defect, not a documentation nit.** On the ordinary Re-issue shape —
|
||||
a consumed-but-failed install on a box that still has its `repo_password` file — the box re-applies,
|
||||
`WriteOffboxSecrets` finds the file present and **keeps it**, and the repo password is unchanged. The
|
||||
hub has nonetheless told the customer, in Hungarian, that their recovery escrow is stale and asked
|
||||
them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** → **R-196**.
|
||||
|
||||
**Corroboration against what actually happened on demo-hp yesterday.** The escrow went `pending` after
|
||||
the Re-issue and the task asks: if the repo password was untouched, what explains that? **It is fully
|
||||
explained, and not by the Re-issue.** Two independent mechanisms, both hub-side-invisible:
|
||||
|
||||
1. `ApplyOffsiteTarget` (`offbox.go:482-494`) carries `EscrowState` over from the *existing* target —
|
||||
but the rebuilt guest had **no existing target**, so `cur` was nil, so the fallthrough
|
||||
`if tgt.EscrowState != "escrowed" { tgt.EscrowState = "pending" }` fired.
|
||||
2. The box had minted a **new** repo password moments earlier (Q2), so `pending` was *correct* — the
|
||||
new key genuinely was not escrowed.
|
||||
|
||||
**The Re-issue's `escrow_stale` event fired at 07:11:51 for the wrong reason and happened to be true
|
||||
for a different one.** Coincidence, not correctness. The proof that these are independent is
|
||||
**demo-felhom**: it received **no Re-issue at all** (measured — its only `escrow_stale` /
|
||||
`offsite_reissued` events in the entire history are dated `2026-07-21 08:29:29`), its repo password
|
||||
changed anyway, and **nothing marked its escrow stale for thirteen hours.**
|
||||
|
||||
> **The mechanism is exactly inverted.** `escrow_stale` is wired to the one path that does **not**
|
||||
> change the repo password, and absent from the path that **does**.
|
||||
|
||||
---
|
||||
|
||||
## Q5 — Why did one box recover itself and the other not?
|
||||
|
||||
### **RULING: confirmed from the ledger — an unconsumed staged secret, thirteen days old. And the recovery was partial in a way the narrative missed.**
|
||||
|
||||
**Method — I4, the `one_time_secrets` ledger, no values read:**
|
||||
|
||||
| customer | `created_at` | `consumed_at` |
|
||||
|---|---|---|
|
||||
| `demo-felhom` | `2026-07-21 08:29:29` | `2026-08-03 07:17:58` |
|
||||
| `demo-hp` | `2026-08-04 07:11:51` | `2026-08-04 07:12:06` |
|
||||
|
||||
demo-felhom's row was **created 2026-07-21 and sat UNCONSUMED for 13 days** — traceable to a Re-issue
|
||||
at that exact timestamp (its `offsite_reissued` + `escrow_stale` event pair, `2026-07-21 08:29:29`).
|
||||
When the rebuild dropped its target on 2026-08-03, the fresh controller consumed the spare at 07:17:58
|
||||
and was reporting `offsite` again by 07:19:10. **76 seconds, on a credential nobody had planned to
|
||||
leave there.** Confirmed from the ledger, not the narrative.
|
||||
|
||||
**What would have happened without one:** exactly demo-hp — `consume` → **404** (`api/offsite.go:20-22`,
|
||||
`no unconsumed offsite password`), the apply-bridge returns
|
||||
`offsite-apply: consume one-time password: …` and retries forever, and the tier stays absent until an
|
||||
operator acts. demo-hp sat there **25 hours** (`2026-08-03 06:12:19` → `2026-08-04 07:12:01`).
|
||||
|
||||
**The correction to the contrast (see Q2):** demo-felhom's spare restored *delivery* only. Its 36
|
||||
snapshots / 1.14 GB were orphaned by the same rebuild. **Neither box's data survived; one box's
|
||||
plumbing did.**
|
||||
|
||||
*Not established:* demo-felhom's escrow generation before 2026-07-21. `host_escrow_superseded` holds
|
||||
only two rows in the whole database (ids 3 and 4, one per box, both superseded 2026-08-04), so whether
|
||||
the 2026-07-21 Re-issue's ceremony re-sealed an *unchanged* password — the predicted false-staleness
|
||||
shape — **cannot be shown from this data**. Stated as unknown rather than asserted.
|
||||
|
||||
---
|
||||
|
||||
## Q6 — Why is delivery one-shot at all?
|
||||
|
||||
### **RULING: the design's reason is that ① is a LIVE PROVIDER PASSWORD, not a bootstrap token; and the honest answer to the security question is that an automatic restage weakens EXPOSURE WINDOW, not authentication.**
|
||||
|
||||
**The design's own reasons, from source:**
|
||||
|
||||
- `api/offsite.go:9-12` — *"serves the one-time transient offsite password to the controller **EXACTLY
|
||||
ONCE** … The value is returned once then marked consumed — a second call 404s. NEVER logged."*
|
||||
- `offsite.go:145-149` — *"the **EXPLICIT** operator recovery for a consumed-password dead-end … It is
|
||||
**NOT** implicit rotation: ProvisionOffsite never calls this."*
|
||||
- `offsiteapply.go:1-6` — the apply-bridge is *"idempotent (a descriptor hash marker prevents
|
||||
**re-consuming a spent password**)"*, i.e. the design treats a wasted consume as a real hazard, which
|
||||
R-71a's whole settle-gate exists to prevent.
|
||||
|
||||
**The security half, stated rather than skipped.** The question is: *could a machine that is not the
|
||||
real one obtain a credential via an automatic restage?*
|
||||
|
||||
**What stands there today is authentication, and a restage does not remove it.** The consume endpoint
|
||||
is gated by `checkAuthCustomer` (`api/offsite.go:14-18`) — **the customer's API key, the same
|
||||
credential as the config pull**. Anything that could consume a restaged password can already pull the
|
||||
customer's whole config. So an automatic restage **grants no new access to an unauthenticated party**.
|
||||
|
||||
**What it genuinely weakens is the exposure window.** Today a live Hetzner sub-account password is
|
||||
fetchable only in the seconds after an operator deliberately staged one. Under an automatic restage it
|
||||
becomes fetchable **whenever the box reports no offsite target** — a condition an attacker holding the
|
||||
API key can arguably induce. The credential's blast radius at the provider is the customer's own
|
||||
sub-account (label-scoped; `ReissueCredentials` refuses unless the label lookup finds **exactly one**,
|
||||
`offsite.go:164,180`), so this is a widening of *when*, not of *what*.
|
||||
|
||||
**And one further guard must survive any change** — R-39(a), `monitor/offsite_delivery.go:26-30`:
|
||||
`SaveOneTimeSecret` is last-write-wins, so restaging on top of an **unconsumed** secret clobbers a
|
||||
password a box may be about to consume. Any automatic restage must re-read the row immediately before
|
||||
acting and refuse unless it is still a **consumed** row. That guard is correct and is not what R-192 is
|
||||
about.
|
||||
|
||||
---
|
||||
|
||||
## Q7 — R-192's guard: exactly what it reads, and why it read wrong
|
||||
|
||||
### **RULING: both halves confirmed, with numbers. The guard reads the OLDEST 500 reports since the consume, and for demo-hp all 500 predated the rebuild by six days.**
|
||||
|
||||
**The query, quoted** (`hub/internal/store/store.go:987`):
|
||||
|
||||
```sql
|
||||
SELECT report_json FROM reports WHERE customer_id = ? AND received_at > ? ORDER BY id LIMIT 500
|
||||
```
|
||||
|
||||
`ORDER BY id` ascending with `LIMIT 500` = **the oldest 500**, not the newest.
|
||||
|
||||
**Reproduced against the live data (I1)**, with demo-hp's real consume anchor `2026-07-23 09:53:41`
|
||||
(recovered from R-192's record — it is no longer in `one_time_secrets`, see I4):
|
||||
|
||||
| measurement | value |
|
||||
|---|---|
|
||||
| `total` (what the guard sees) | **500** |
|
||||
| `withOffsite` (what the guard sees) | **500** |
|
||||
| oldest report in that window | `2026-07-23 09:53:47` |
|
||||
| newest report in that window | **`2026-07-28 11:17:40`** |
|
||||
| true `total` over the same window, no LIMIT | 1174 |
|
||||
| true `withOffsite` over the same window, no LIMIT | 1063 (⇒ 111 without — matching the 111 ABSENT reports in I3) |
|
||||
|
||||
**The whole 500-report evidence set ends 2026-07-28 — six days before the 2026-08-03 rebuild.**
|
||||
|
||||
**Defect (a) — the message is exactly inverted.** `maybeEmitStuck`
|
||||
(`monitor/offsite_delivery.go:~110`) interpolates `status.ReportsSinceConsume` into a string that
|
||||
hardcodes the phrase *"carry no offbox target"*, and never reads `OffsiteReportsSinceConsume`. The
|
||||
live e-mail, quoted in full from the events table:
|
||||
|
||||
> *"Offsite delivery stuck: one-time password consumed 284h19m0s ago and **500 report(s) since carry
|
||||
> no offbox target** — the credential is likely burned (apply died between consume and persist).
|
||||
> Re-issue delivers a fresh one."*
|
||||
|
||||
`OffsiteReportsSinceConsume` for that same call was **500**. Every one of them carried a target. The
|
||||
message states the precise negation of its own measurement, and prescribes a remedy for a failure mode
|
||||
that did not occur.
|
||||
|
||||
**Defect (b) — the heal refuses silently, on stale evidence.** `maybeHeal` returns bare on
|
||||
`status.OffsiteReportsSinceConsume != 0` (it was 500), with **no log line**. `offsite_credential_restaged`
|
||||
has never fired for any customer — confirmed: zero rows of that event type in the database.
|
||||
|
||||
**What the guard SHOULD discriminate on — the shape, not the code.** The guard's intent is right:
|
||||
*"there is offbox evidence, so this is a regressed apply, not a burned credential — the operator's
|
||||
call."* Its error is that **it asks the question of the wrong time period**. Offbox evidence from
|
||||
before a rebuild is not evidence that the credential still works. The discriminator must be
|
||||
**recency-bounded and rebuild-aware**: judge on evidence *after the newest `controller_started` /
|
||||
config-hash change*, or on the latest N reports — never on "everything since the consume". And a
|
||||
refusal must **name its reason in the log**, because "we chose not to heal" and "the heal never ran"
|
||||
must not look identical (the S-16 rule, one layer down).
|
||||
|
||||
**Do not fix it in isolation.** Under Q2's finding, a successful auto-restage on demo-hp would have
|
||||
restored the transport and the box would *still* have minted a new repo password. **The heal cannot
|
||||
protect the data; it can only protect the plumbing.** Whatever shape it takes must say so.
|
||||
|
||||
---
|
||||
|
||||
## Q8 — What each candidate would cost
|
||||
|
||||
**A fourth option exists and is the important one**, because (a) and (b) both address ① and Q2 shows
|
||||
the damage is in ③.
|
||||
|
||||
### (a) The hub restages automatically when a re-enrolled box reports no offsite
|
||||
|
||||
**A version of this ALREADY EXISTS and is wired to the wrong event.** `reissueOnReenroll`
|
||||
(`hub/internal/api/handler.go:1051-1084`), leg **F3**:
|
||||
|
||||
> *"F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
||||
> one-time password only ever reached the OLD controller)."*
|
||||
|
||||
It is called from `handleHostEnroll` — but **after** a mint-once-reuse short-circuit
|
||||
(`handler.go:1004-1016`): `if existing != nil { …return… }`. **A guest rebuild leaves the `hosts` row
|
||||
intact** (measured: `demo-hp-bb76ea` is the same host_id throughout the incident; only the agent leaf
|
||||
re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the existing credential and
|
||||
**F3 is never reached.**
|
||||
|
||||
- **Changes:** hub only. Either lower the trigger from host-enrollment to a controller-level rebuild
|
||||
signal (the `config_hash` change + `controller_started` pair the hub already receives and already
|
||||
logs), or fix R-192's guard so the existing R-71c self-heal can act.
|
||||
- **Weakens:** the exposure window in Q6. Needs the R-39(a) unconsumed-secret guard kept intact.
|
||||
- **Does NOT solve:** ③. The box still mints a new repo password and still orphans the repo. **This
|
||||
option restores the plumbing to a box whose data key is already gone.**
|
||||
- **Cost:** small. **Value against the actual harm: near zero.**
|
||||
|
||||
### (b) The credential becomes recoverable from escrow at re-bootstrap
|
||||
|
||||
- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
|
||||
**R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
|
||||
(`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
|
||||
customer's recovery code.** Any design that says "recover it at re-bootstrap" is describing a
|
||||
customer-present ceremony, i.e. what the existing manual `offboxInjectPasswordHandler` already is.
|
||||
- **Does NOT solve:** an unattended rebuild, which is the whole scenario.
|
||||
- **Cost: not implementable as specified.**
|
||||
|
||||
### (c) — NOT PREVIOUSLY NAMED — the agent retains the repo password across a guest rebuild
|
||||
|
||||
The agent lives **on the Proxmox host** and survives a guest rebuild. It already receives the repo
|
||||
password today: the controller pushes it over the pinned local API
|
||||
(`POST /escrow/stage-secret` → `agent/internal/localapi/escrow_stage.go`), and the agent writes it to
|
||||
the fixed 0600 path `/var/lib/felhom-agent/escrow-stage/restic_repo_password`
|
||||
(`agent/internal/escrow/identity.go:44-46`) **transiently — wiped by the ceremony**
|
||||
(`WipeStagedResticPassword`). The seam, the transport, the pinning and the file are all already built.
|
||||
|
||||
- **Changes:** agent (retain rather than wipe; serve back over the same pinned local API) + controller
|
||||
(on a fresh data dir, ask the agent before minting — the `InjectOffboxPassword` seam already exists
|
||||
and is exactly the right shape).
|
||||
- **Weakens:** the escrow's *zero-knowledge* posture in one specific way — the irreplaceable data key
|
||||
would now sit at rest on the Proxmox host, which the operator has root on. **Measured against D6,
|
||||
that is not a new exposure in principle** (D6 already states plainly that *"the operator cannot read
|
||||
customer data"* was never the security property, and the operator holds root on every box) — but it
|
||||
IS a new copy, and copies are the thing R-133 is already open about for the hub DB. **This is a real
|
||||
trade and it is the operator's to make.**
|
||||
- **Solves:** exactly the failure that occurred, unattended, on every rebuild, without any hub
|
||||
involvement and without any credential leaving the premises.
|
||||
- **Cost:** medium. Two repos, one existing seam each.
|
||||
|
||||
### (d) Do neither — keep the manual Re-issue, and fix what LIES
|
||||
|
||||
- **Changes:** R-192's message + refusal log; correct the three false comments about what Re-issue
|
||||
rotates; and — the part that actually matters — **make a repo-password change VISIBLE**. The hub
|
||||
already receives `restic_pw_sha256` on every escrow upload and already stores it. Comparing the new
|
||||
hash against the superseded one is a two-line verdict, and it is the signal that was missing on
|
||||
demo-felhom for thirteen hours → **R-197**. (Note the shape: **this spike answered its own hardest
|
||||
question with a comparison the hub could be making automatically and is not.** Both values were
|
||||
already in the database; nothing read them.)
|
||||
- **Weakens:** nothing.
|
||||
- **Does NOT solve:** the data loss. Every rebuild still orphans the repo; the operator simply finds
|
||||
out the same day instead of never.
|
||||
- **Cost:** small.
|
||||
|
||||
### Ranked, with a recommendation
|
||||
|
||||
| rank | option | why |
|
||||
|---|---|---|
|
||||
| **1** | **(d) now — the honesty pass** | It is cheap, it weakens nothing, and **it is the only option that helps the incident already in flight.** The alarm that lies is worse than no alarm; the repo-password-changed detector is the one signal whose absence let demo-felhom lose 1.14 GB of history silently. Ship this regardless of what is decided about the rest. |
|
||||
| **2** | **(c) — agent-retained repo password** | The only candidate that addresses ③, which is where the harm is. Every seam it needs exists. It costs one deliberate trade-off the operator must actually make. |
|
||||
| **3** | **(a) — auto-restage** | Worth doing **after** (c), never instead of it. On its own it restores the transport to a box whose data key is already gone, and would have made both boxes look healthy on 2026-08-04 while the snapshots were orphaned — a *strictly worse* outcome than the current loud failure. |
|
||||
| **4** | **(b) — recover from escrow at re-bootstrap** | Not implementable as stated; the escrow is zero-knowledge by design. |
|
||||
|
||||
**Recommendation: ship (d), then decide (c).** And do not ship (a) first — it would have hidden this.
|
||||
|
||||
---
|
||||
|
||||
## What could not be established
|
||||
|
||||
Named rather than guessed (§8 rule 4).
|
||||
|
||||
1. **The 2026-08-05 02:15 UTC run's actual outcome** (Q3). Predicted `ErrOffboxOrphaned` from source +
|
||||
claim state; **not measured**.
|
||||
2. **Whether the orphaned ciphertext still exists at the provider.** 40.9 MB (demo-hp) + 1.14 GB
|
||||
(demo-felhom) are *presumed* present at `/home/felhom-repo` — reading the endpoint needs the SFTP
|
||||
credential, which this session did not touch. The Hetzner API has no directory-listing surface.
|
||||
3. **Whether demo-felhom's 2026-07-21 Re-issue re-sealed an unchanged password** (the predicted
|
||||
false-staleness shape, Q4/Q5). Only two `host_escrow_superseded` rows exist in the whole database;
|
||||
the generation before 2026-07-21 is not retained.
|
||||
4. **What removed demo-hp's offsite target at the rebuild** is fully explained (fresh data dir), but
|
||||
**why the guest was rebuilt on both boxes on consecutive mornings** is outside this spike.
|
||||
|
||||
---
|
||||
|
||||
## The operator's question — stated, and NOT answered here
|
||||
|
||||
> **The irreplaceable secret in the offsite tier is the restic repository password. It is generated on
|
||||
> the box, it is destroyed by a guest rebuild, and the only copy that survives is sealed under the
|
||||
> customer's recovery code — which nothing but a human can open. Every machine is going to be
|
||||
> reinstalled.**
|
||||
>
|
||||
> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, where
|
||||
> you already have root — or do you want it to stay sealed under the customer's recovery code and
|
||||
> accept that every rebuild starts a new repository and orphans the old one, provided the system says
|
||||
> so loudly on the day it happens?**
|
||||
|
||||
Option (c) is the first. Option (d) alone is the second. **This spike does not choose.**
|
||||
File diff suppressed because one or more lines are too long
@@ -1,3 +1,48 @@
|
||||
## v0.92.0 — a customer with no machine ever bound does not alarm (2026-08-04, R-195)
|
||||
|
||||
`david` is a real prospective customer whose record was created 2026-08-01 with **no host ever bound**
|
||||
— `hosts=0`, `host_deletions=0`, `host_reports=0`, `reports=0` — and it e-mailed an
|
||||
`expected_dbdump_missed` **error** at 03:00 UTC on 2026-08-02, 08-03 and 08-04. Nothing has ever been
|
||||
expected of that customer, so the honest verdict is UNKNOWN, not missed.
|
||||
|
||||
**The mechanism is the interesting half, because the guard that should have covered this is keyed off
|
||||
the wrong thing.** `CheckBackupDeadlines` already skips nodes that are `down`, and that skip reads
|
||||
`StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — **a query over the
|
||||
`reports` table**. A customer with zero reports is in no row, gets no state at all, and `GetState()`
|
||||
returns `""` rather than `"down"`. **The skip misses exactly the customer it would most obviously
|
||||
cover.** Corroborated on live data: `peti-felhom` is active with a host deleted 2026-07-15 and does
|
||||
*not* alarm, because it has 482 old reports and is therefore `down`. The backup half was already safe
|
||||
(`reportJSON == ""` → skip); the DB-dump half had no guard at all.
|
||||
|
||||
- **`store.HasEverBoundHost(customerID)`** — a live `hosts` row **OR** a `host_deletions` tombstone.
|
||||
Consulted once per customer at the top of the deadline loop; a false verdict skips the customer with
|
||||
an INFO line and its own counter in the summary.
|
||||
- **The discriminator is "was a host EVER bound", NOT "has a report arrived", and that is the whole
|
||||
design.** A box that was installed, bound and then went silent is a real fault and must keep
|
||||
alarming — it has a `hosts` row, so it is judged. Only a customer that never had a machine is
|
||||
UNKNOWN. The tombstone is included for the same reason: a customer whose host was deleted *had* one,
|
||||
and that judgement belongs to the staleness checker, not to this predicate.
|
||||
- **Fail-OPEN on a read error.** An unreadable binding must never SUPPRESS a real alarm; it logs and
|
||||
judges anyway.
|
||||
- **The anchored-verdict structure is untouched** (R-81/v0.73.0). This is the same invariant — absence
|
||||
is UNKNOWN until an anchor elapses — applied one level up, at the question of whether there is a
|
||||
subject at all.
|
||||
|
||||
**Tests, including the one that matters.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`
|
||||
is a real shape — a machine installed and never phoned home — and it is indistinguishable from `david`
|
||||
on every signal except the one the guard reads. If the guard is ever "simplified" to key off report
|
||||
presence, customer age or a name pattern, it goes red. Plus a bound-then-went-quiet case, a
|
||||
deleted-host case, and the predicate's own three inputs. **Companion red-proof observed:** removing the
|
||||
guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` —
|
||||
verbatim the event `david` sent three mornings running. Restored after.
|
||||
|
||||
**`david`'s record was not modified.** The record is correct; the alarm was what was wrong.
|
||||
|
||||
**Shipped alongside a spike that deliberately shipped NO code** —
|
||||
`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md` (R-193/R-192): what is one-shot
|
||||
in the offsite path, and what a guest rebuild really costs. It ends in ranked options and a question
|
||||
for the operator; R-196 and R-197 were minted from it and are not implemented here.
|
||||
|
||||
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
|
||||
|
||||
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
|
||||
|
||||
@@ -336,7 +336,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
|
||||
sinceUTC := midnightBudapest.UTC()
|
||||
|
||||
var backupMissed, dbdumpMissed, skipped, deferred int
|
||||
var backupMissed, dbdumpMissed, skipped, deferred, unbound int
|
||||
|
||||
for _, id := range customerIDs {
|
||||
// Skip nodes that are down — they already have staleness events
|
||||
@@ -350,6 +350,37 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
continue
|
||||
}
|
||||
|
||||
// ── R-195: a customer with NO machine EVER bound is UNKNOWN, not missed ────────────────
|
||||
//
|
||||
// Both verdicts below ask "did the thing we expect every day happen?". For a customer
|
||||
// that has never had a machine bound, nothing has ever been expected, so the honest
|
||||
// answer is UNKNOWN — the same invariant assessBackupFreshness states above, applied one
|
||||
// level up, at the question of whether there is a subject at all.
|
||||
//
|
||||
// The discriminator is "was a host EVER bound", NOT "has a report arrived". That is the
|
||||
// case this check must not break: a box that was installed, bound, and then went silent
|
||||
// has a real fault and must keep alarming. It is bound, so it is judged.
|
||||
//
|
||||
// WHY THIS WAS REACHABLE AT ALL, measured 2026-08-04: the down-skip above is what
|
||||
// protects every other silent customer, and it reads the staleness checker's state — which
|
||||
// is seeded from the `reports` table (store.GetCustomers). A customer that has NEVER
|
||||
// reported appears in no report row, so it gets no staleness state at all and GetState()
|
||||
// returns "" rather than "down". The skip misses exactly the customer it would most
|
||||
// obviously cover, and the DB-dump half below then fires every night: `david`, a
|
||||
// prospective customer whose record was created 2026-08-01 with no machine ever bound,
|
||||
// e-mailed an expected_dbdump_missed ERROR at 03:00 UTC on three consecutive days.
|
||||
//
|
||||
// Fail-open on a read error: an unreadable binding must never SUPPRESS a real alarm.
|
||||
if bound, berr := s.HasEverBoundHost(id); berr != nil {
|
||||
logger.Printf("[WARN] Deadline check: failed to read host binding for %s (judging anyway): %v", id, berr)
|
||||
} else if !bound {
|
||||
// Visible, per the v0.73.0 Part-7 precedent below: a quiet check must never be
|
||||
// indistinguishable from a check that did not run. Once daily, one line per customer.
|
||||
logger.Printf("[INFO] Deadline check: %s has no host EVER bound — all deadline verdicts UNKNOWN (no alarm)", id)
|
||||
unbound++
|
||||
continue
|
||||
}
|
||||
|
||||
// Backup freshness from the agent's host-report (PBS snapshots + vzdump),
|
||||
// the authoritative offsite-backup signal post-slice-8C.
|
||||
reportJSON, rerr := s.GetLatestHostReportJSON(id)
|
||||
@@ -423,6 +454,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
}
|
||||
}
|
||||
|
||||
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)",
|
||||
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped)
|
||||
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down), %d unknown (no host ever bound)",
|
||||
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped, unbound)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-195 — a customer with NO machine EVER bound must not alarm; a customer WITH one must.
|
||||
//
|
||||
// Origin, measured on the live hub 2026-08-04: `david` is a prospective customer whose record was
|
||||
// created 2026-08-01 16:51:49 with no host ever bound (hosts=0, host_deletions=0, host_reports=0,
|
||||
// reports=0). It e-mailed an `expected_dbdump_missed` ERROR at 03:00 UTC on 08-02, 08-03 and 08-04.
|
||||
//
|
||||
// The mechanism, established at source: the down-skip in CheckBackupDeadlines reads
|
||||
// StalenessChecker.GetState(), whose map is seeded from store.GetCustomers() — a query over the
|
||||
// `reports` table. A customer with zero reports is in no row, so it has no state, and GetState()
|
||||
// returns "" rather than "down". The skip that protects every other silent customer misses the one
|
||||
// that never reported at all.
|
||||
//
|
||||
// These tests pin BOTH halves. A suite that only proved the silence would pass against an
|
||||
// implementation that never alarms, which is strictly worse than the defect it replaces.
|
||||
|
||||
// newUnboundStore creates a store holding ONE active customer and NO host row at all.
|
||||
func newUnboundStore(t *testing.T) *store.Store {
|
||||
t.Helper()
|
||||
st, err := store.New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatalf("store.New: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"}); err != nil {
|
||||
t.Fatalf("SaveCustomerConfig: %v", err)
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// ── Half 1: the silence ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// TestCheckBackupDeadlines_NeverBoundHost_Silent is the david case.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): deleting the HasEverBoundHost guard from CheckBackupDeadlines
|
||||
// makes this test fail with
|
||||
//
|
||||
// deadline_unbound_test.go: a customer with NO host ever bound must raise NOTHING;
|
||||
// got [expected_dbdump_missed]
|
||||
//
|
||||
// which is verbatim the event `david` e-mailed three mornings running. Restored after.
|
||||
func TestCheckBackupDeadlines_NeverBoundHost_Silent(t *testing.T) {
|
||||
st := newUnboundStore(t)
|
||||
// No UpsertHost, no host-report, no db_dump_completed event — nothing has ever been expected.
|
||||
got := runDeadline(t, st)
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("a customer with NO host ever bound must raise NOTHING; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHasEverBoundHost pins the predicate itself across its three inputs, because the whole
|
||||
// behaviour above turns on it and a predicate that answered `false` for everything would make the
|
||||
// test above pass while silencing the entire fleet.
|
||||
func TestHasEverBoundHost(t *testing.T) {
|
||||
st := newUnboundStore(t)
|
||||
|
||||
if bound, err := st.HasEverBoundHost("c1"); err != nil || bound {
|
||||
t.Fatalf("no host rows → want (false,nil); got (%v,%v)", bound, err)
|
||||
}
|
||||
|
||||
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||
t.Fatalf("UpsertHost: %v", err)
|
||||
}
|
||||
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
|
||||
t.Fatalf("live host row → want (true,nil); got (%v,%v)", bound, err)
|
||||
}
|
||||
|
||||
// An unknown customer is never bound — the predicate must not answer from another customer's rows.
|
||||
if bound, err := st.HasEverBoundHost("nobody"); err != nil || bound {
|
||||
t.Fatalf("unknown customer → want (false,nil); got (%v,%v)", bound, err)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Half 2: THE RED-PROOF THAT MATTERS — a bound machine must still alarm ───────────────────────
|
||||
|
||||
// TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms is the case the change could break,
|
||||
// and it is a real shape: a machine that was installed and bound and never phoned home. It has a
|
||||
// `hosts` row and zero reports — indistinguishable from `david` on every signal EXCEPT the one the
|
||||
// guard discriminates on. If the guard is ever "simplified" to key off report presence, customer
|
||||
// age or a name pattern, this test goes red.
|
||||
func TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms(t *testing.T) {
|
||||
st := newUnboundStore(t)
|
||||
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||
t.Fatalf("UpsertHost: %v", err)
|
||||
}
|
||||
// No host-report and no db_dump_completed — the box was bound and never said anything.
|
||||
got := runDeadline(t, st)
|
||||
if !has(got, "expected_dbdump_missed") {
|
||||
t.Fatalf("a BOUND machine that never reported is a real fault and must still alarm; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms: the machine reported once, days ago,
|
||||
// and stopped. Nothing about the guard may suppress that.
|
||||
//
|
||||
// The staleness checker's down-skip is nil here (runDeadline passes nil), which is deliberate: it
|
||||
// isolates THIS guard. In production a genuinely down node is skipped by staleness and gets its own
|
||||
// node_down event — that path is unchanged and is not what this test is about.
|
||||
func TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms(t *testing.T) {
|
||||
st := newUnboundStore(t)
|
||||
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||
t.Fatalf("UpsertHost: %v", err)
|
||||
}
|
||||
// One host-report whose newest backup evidence is 9 days old, and a db dump that last
|
||||
// completed 5 days ago (i.e. not since midnight).
|
||||
report := hostReportJSON(t, [][2]string{{rfc(-9 * 24 * time.Hour), "ok"}}, nil)
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatalf("SaveHostReport: %v", err)
|
||||
}
|
||||
got := runDeadline(t, st)
|
||||
if !has(got, "expected_backup_missed") {
|
||||
t.Fatalf("a bound machine that went quiet with stale backups must still raise expected_backup_missed; got %v", got)
|
||||
}
|
||||
if !has(got, "expected_dbdump_missed") {
|
||||
t.Fatalf("a bound machine that went quiet must still raise expected_dbdump_missed; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCheckBackupDeadlines_DeletedHost_StillJudged: the host row is gone but a tombstone remains
|
||||
// (peti-felhom's live shape). The customer HAD a machine, so this check must not take over the
|
||||
// judgement — it hands off to the staleness down-skip exactly as before the change.
|
||||
func TestCheckBackupDeadlines_DeletedHost_StillJudged(t *testing.T) {
|
||||
st := newUnboundStore(t)
|
||||
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||
t.Fatalf("UpsertHost: %v", err)
|
||||
}
|
||||
if err := st.DeleteHost("h1", false); err != nil {
|
||||
t.Fatalf("DeleteHost: %v", err)
|
||||
}
|
||||
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
|
||||
t.Fatalf("a DELETED host is still a machine that was once bound → want (true,nil); got (%v,%v)", bound, err)
|
||||
}
|
||||
got := runDeadline(t, st)
|
||||
if !has(got, "expected_dbdump_missed") {
|
||||
t.Fatalf("a customer whose host was deleted was still bound and stays judged here; got %v", got)
|
||||
}
|
||||
}
|
||||
@@ -2262,6 +2262,28 @@ func (s *Store) GetHostByCustomer(customerID string) (*Host, error) {
|
||||
return h, err
|
||||
}
|
||||
|
||||
// HasEverBoundHost reports whether a machine was EVER bound to this customer — a live row in
|
||||
// `hosts` OR a tombstone in `host_deletions`. It answers "was anything ever expected of this
|
||||
// customer", which is the question the deadline verdicts actually need (R-195).
|
||||
//
|
||||
// It is deliberately NOT "has a report arrived", and the distinction is the whole point: a box
|
||||
// that was installed, bound, and then went silent IS bound, and its silence is a real fault that
|
||||
// must keep alarming. Only a customer that never had a machine at all is UNKNOWN.
|
||||
//
|
||||
// `host_deletions` is included because a customer whose host was removed HAD one — the deadline
|
||||
// caller reaches its down-skip for that shape, and this predicate must not quietly take over a
|
||||
// judgement the staleness checker owns.
|
||||
func (s *Store) HasEverBoundHost(customerID string) (bool, error) {
|
||||
var n int
|
||||
if err := s.db.QueryRow(
|
||||
`SELECT EXISTS(SELECT 1 FROM hosts WHERE customer_id = ?)
|
||||
OR EXISTS(SELECT 1 FROM host_deletions WHERE customer_id = ?)`,
|
||||
customerID, customerID).Scan(&n); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return n != 0, nil
|
||||
}
|
||||
|
||||
// ListHostsByCustomer returns the customer's hosts ordered by host_id (v0.47.0 — the
|
||||
// customer page's Host tab is a LIST by design: 1 host today, N for a later HA cluster).
|
||||
// Uses the idx_hosts_customer index.
|
||||
|
||||
Reference in New Issue
Block a user