docs: R-196 closed, R-204 items 1-3 closed, item 4 open (R-193)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
- OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and its dependency stated. Header restates that R-202, the 1.2 GB ciphertext deletion and R-198's still-unit-proven retention all REMAIN OPEN. - capability map: the recovery row keeps its 'with a person present' qualifier, names which crutch remains, and cites the three now gone. - 07-backup-architecture: new 7.0 - what a customer can and cannot do ALONE, the four steps in a table with status. This is the section a future reader will use to answer that question. - CONTEXT: standing ruling S-32, superseding S-31 steps 2-5. - STATUS: rewritten to one screen per its own header; removes a corrupted half-overwritten section left from the drill session. - ROADMAP: R-196 and R-204 collapsed.
This commit is contained in:
@@ -1,124 +1,160 @@
|
||||
# REPORT — R-201: **THE DRILL PASSED.** A customer's file survived a machine rebuild
|
||||
# REPORT — R-196 / R-204 item 2 (hub v0.95.0), 2026-08-05
|
||||
|
||||
**2026-08-04, 21:30–23:15** · `demo-hp` deliberately rebuilt · no code, no version bump ·
|
||||
`demo-felhom` untouched. Record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.
|
||||
**A re-issue no longer marks a healthy escrow stale.** One behaviour change, one register closed, and
|
||||
the coverage claim proved rather than assumed. The controller's half of R-204 (items 1 and 3) is
|
||||
`felhom-controller` v0.198.0.
|
||||
|
||||
---
|
||||
## 1. Baselines, re-read on arrival
|
||||
|
||||
## 1. THE VERDICT — PASS
|
||||
| Repo | Expected | Found |
|
||||
|---|---|---|
|
||||
| `felhom.eu` | `2a7ac03c4726` / hub v0.94.0 (deployed `felhom-hub:0.94.0`) | **exact match**, tree clean, `HEAD == origin/main` |
|
||||
|
||||
```
|
||||
pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||
restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||
```
|
||||
**§3.2's landmark had DRIFTED, and the drift changed the work.** The task described
|
||||
`offsite.go:222-231` under a known-consequence comment saying the mark was made on a false premise.
|
||||
That comment had already been rewritten by the R-196 comment-correction commit, and the version on
|
||||
`main` gave a **non-false** ground for the mark: *"the box's re-apply may mint a fresh repository
|
||||
password (it does exactly that whenever `<DataDir>/offbox/repo_password` is absent — the
|
||||
guest-rebuild shape)"*. So the question was no longer "delete a comment's lie" but "is the shape it
|
||||
guards actually covered elsewhere?" — which is Scenario D, and §8.2 says to stop and report if it is
|
||||
not. It is; §3 below is the evidence.
|
||||
|
||||
**Byte-identical.** The controller's data volume was destroyed and the file deleted from disk; the
|
||||
customer's recovery code, the customer's restore flow, and the file came back. **The Felhom backup
|
||||
story is proved end to end for the first time.**
|
||||
## 2. What changed
|
||||
|
||||
## 2. Snapshot count at step 9 — **3, not 1**
|
||||
`offsite.ReissueCredentials` no longer calls `MarkEscrowStale` and no longer emits the `escrow_stale`
|
||||
event. **`offsite_reissued` is untouched** and still fires on every re-issue. The known-consequence
|
||||
comment is rewritten to record what was done, when, and why — with the disagreement below stated in
|
||||
it rather than absorbed.
|
||||
|
||||
`repo_state: null` (the repository **opened**, not orphaned), `snapshot_count: 3`, `repo_size_bytes:
|
||||
42 026` — the pre-wipe size exactly. The pre-wipe snapshot `e6132ae5` was still present with the file
|
||||
inside it. **No fresh history was started over the old one** — the failure this drill was designed to
|
||||
catch did not occur.
|
||||
**What the mark actually cost, established mechanically rather than asserted** (this is why it was a
|
||||
blocker and not a nit):
|
||||
|
||||
## 3. §5's five conditions, recorded before the wipe
|
||||
1. `stale_at` set → `GetEscrowStatusForCustomer` **withholds** `restic_pw_sha256` from the report ACK.
|
||||
2. With no hash, the controller's SLICE-3 auto-confirm returns early and cannot flip
|
||||
`pending → escrowed`.
|
||||
3. `OffboxRunnable() = OffboxConfigured() && EscrowState == "escrowed"` → **every off-site backup
|
||||
refused**, indefinitely, on a box whose repository key was never in doubt.
|
||||
4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
|
||||
supersedes the sealed blob. **During a recovery that is the one act that would have destroyed the
|
||||
key just recovered.**
|
||||
|
||||
1. sentinel **listed by name** in snapshot `e6132ae5` (19:36:26), 181 B.
|
||||
2. rollback archive `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` (1 606 765 083 B) **verified by a
|
||||
full zstd stream read** (4 867 573 760 B) with the sentinel confirmed inside.
|
||||
3. §3's option — §4 below.
|
||||
4. `demo-felhom` `health=ok`, `escrow_state=escrowed`, untouched throughout.
|
||||
5. nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.
|
||||
## 3. Scenario D — the evidence that the removed marking is covered
|
||||
|
||||
**One precondition had drifted and was repaired, not worked around:** the staged snapshot had lost the
|
||||
sentinel because the afternoon's experiment produced a *later* same-day snapshot and `forget
|
||||
--keep-daily 7 --group-by host,tags` pruned the good one. One backup re-established it, re-verified by
|
||||
listing. **A good snapshot is not durable against a later bad run on the same day.**
|
||||
The mark was precautionary and aimed at ONE shape: a re-issue followed by a box that mints a fresh
|
||||
repository password (the guest-rebuild shape, where `offbox/repo_password` is absent). That shape is
|
||||
measured in two independent places, and **the mark was blinding one of them**:
|
||||
|
||||
## 4. §3 — the recovery code
|
||||
- **Continuous, box-side — the real coverage.** `report.EscrowAutoConfirmer.reconcileEscrowed`
|
||||
(controller) compares the ACK's sealed `restic_pw_sha256` against the box's CURRENT local repo
|
||||
password on **every report ACK**, raising the stale flag, the customer card and the
|
||||
„create a new recovery code" CTA on a mismatch. That is a **measurement**, not a guess, and it is
|
||||
continuous rather than edge-triggered. Pinned by the controller's
|
||||
`TestEscrowStale_MismatchWarnsOnceAndFlags` — re-run green this session.
|
||||
**And step 1 above was blinding it:** a stale flag empties the very hash that comparison needs, so
|
||||
the box could only reach the hash-LESS branch and report *"the hub's current blob carries NO
|
||||
password hash"* — which is false. Removing the mark restores the true signal.
|
||||
- **Edge-triggered, hub-side.** R-197's `offsite_repo_key_changed` fires from
|
||||
`api.maybeEmitRepoKeyChanged` on a proven hash difference across a supersession and pages the
|
||||
operator. **Red-proved:** removing the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut`
|
||||
made `TestEscrowPut_ChangedRepoKey_RaisesSignal` fail with *"the repository key demonstrably changed
|
||||
and NO signal was raised"*, while the two silence tests stayed green.
|
||||
|
||||
**Option B as already in place, improved: no new copy was made, so nothing needed shredding.** The
|
||||
operator's own `~/.config/credentials` (`0600`) was the source; the code was **piped to stdin** for
|
||||
each invocation — never an argument, never exported, never a second file, never logged. Destroying the
|
||||
operator's own store would have destroyed their record; because no extra copy existed, there is nothing
|
||||
left to prove gone.
|
||||
**Disagreement recorded, per the R-96 standing rule.** Scenario D as written asks that a real key
|
||||
change also *"mark the escrow stale"*. **It must not, and nothing was changed to make it.** The hub
|
||||
learns of a real change at the instant a supersession **seals the new password** — i.e. when the
|
||||
escrow is at its freshest. Marking it stale there would ask the customer for a ceremony to fix the
|
||||
ceremony that just ran. The correct consequence at that instant is the operator alarm, which is
|
||||
exactly what R-197 already does. This is recorded in the code comment, the CHANGELOG and OPEN-ITEMS,
|
||||
not only here.
|
||||
|
||||
**Verified anyway, with the planted-copy positive control:** 0 hits in the agent journal, 0 in the
|
||||
controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover
|
||||
`felhom-idesc-*` dirs — and the same sweep found a planted copy (**1**), then **0** after shredding it.
|
||||
## 4. `MarkEscrowStale` is kept with no caller — deliberately
|
||||
|
||||
## 5. Every step's observable
|
||||
Per task §5 it was not to be modified, and it is not deleted either. The `stale_at` flag remains live
|
||||
and correct — read by the ACK, the operator config card and the PBS-DR view — and the right way to
|
||||
set it is a **future EVIDENTIAL caller** that has measured a key change rather than guessed at one.
|
||||
Its doc comment now says so plainly instead of naming a caller that no longer exists, and
|
||||
`TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying to inert while
|
||||
nothing writes it (the seam-built-but-never-wired shape, in reverse).
|
||||
|
||||
| step | observable |
|
||||
The schema comment and `EscrowStatus.Stale`'s comment were corrected the same way — each of the three
|
||||
previously asserted a writer that is now gone.
|
||||
|
||||
## 5. Files modified
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| 6 | fresh data dir `20:00:2x`; new `encryption.key`; `claimed = None`; `offbox = null` |
|
||||
| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED.** Nothing re-escrowed itself |
|
||||
| **8** | **recovered `8a9e33aa4da6…`** — matches the pre-wipe on-disk key and the hub's record |
|
||||
| 9a | `[INSTALLED] … reads back identical` |
|
||||
| 9b | after Re-issue + apply, the on-disk key is **still** the recovered one |
|
||||
| 9c | **repository OPENED** — `repo_state: null`, 3 snapshots, 42 026 B |
|
||||
| **10** | **restored sha256 byte-identical** |
|
||||
| 11 | this record |
|
||||
| `hub/internal/offsite/offsite.go` | the pessimistic `MarkEscrowStale` + `escrow_stale` event removed; comment rewritten to record the change, the coverage and the disagreement |
|
||||
| `hub/internal/offsite/offsite_test.go` | `TestReissue_InvalidatesEscrow` **replaced in place by its exact inverse** `TestReissue_DoesNotMarkAHealthyEscrowStale`; new `TestEscrowStaleMechanism_StillWithholdsAndClears` |
|
||||
| `hub/internal/store/store.go` | three comments corrected (`MarkEscrowStale`, the `stale_at` schema note, `EscrowStatus.Stale`) — each had named a writer that no longer exists |
|
||||
| `manifests/hub.yaml` | image tag `0.94.0` → `0.95.0` |
|
||||
| `hub/CHANGELOG.md`, `CONTEXT.md`, `STATUS.md`, `documentation/…` | v0.95.0 entry; ruling **S-32**; the register and architecture updates below |
|
||||
|
||||
Superseded escrow rows remained **2** throughout — **no ceremony was run at any point.**
|
||||
**Commits on `main`:** `d1a8edb` (behaviour + tests + comments) · `5c7d671` (CHANGELOG) ·
|
||||
`975a690` (manifest bump).
|
||||
|
||||
## 6. Part 2 — not run
|
||||
**Deploy:** built + pushed `felhom-hub:0.95.0`, bumped `manifests/hub.yaml`, pushed, then a
|
||||
**deliberate ArgoCD hard-refresh + sync** (auto-sync stays off; no `kubectl set image` anywhere).
|
||||
Result: app `felhom` **Synced / Healthy**, `deploy/hub` rolled out, running image
|
||||
`gitea.dooplex.hu/admin/felhom-hub:0.95.0`, startup log clean (offsite provisioning, pool-box checker,
|
||||
PBS-DR reconciler and all six host checkers initialised; `Listening on :8080`).
|
||||
|
||||
Gate: *"the drill PASSED; its evidence is written down; and there is time."* The first two are met; the
|
||||
third is not — 23:15, after a full destructive cycle, and Part 2 is a second wipe. Deliberately left
|
||||
for a fresh session with drill 1's result already recorded, so a second wipe cannot overwrite it.
|
||||
**R-198's retention remains unit-proven only** — nothing has yet superseded a key in production, and it
|
||||
is now the last unproven link in this chain.
|
||||
## 6. Tests and red-proofs
|
||||
|
||||
## 7. Teardown — three layers
|
||||
Green gate: `cd hub && go build ./... && go vet ./... && go test ./...` — **rc=0**.
|
||||
`python3 scripts/repo_gates.py --fast` — **all five gates OK**.
|
||||
|
||||
| layer | state |
|
||||
|---|---|
|
||||
| the guest | **healthy and re-armed**: controller v0.197.0 claimed, six app containers serving, all three apps re-toggled for off-site, the sentinel restored to its live location and verified. The verification-copy tree is **kept as evidence**. Scratch band empty |
|
||||
| the host | vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted |
|
||||
| the hub | **no new customer records** — `demo-hp` is the same row throughout (a controller-data rebuild, not a re-enrolment). One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. **The two superseded escrow rows are unchanged** |
|
||||
| Test | Result | Red-proof — what was mutated | Outcome |
|
||||
|---|---|---|---|
|
||||
| `TestReissue_DoesNotMarkAHealthyEscrowStale` (C) | PASS | restored the pessimistic `MarkEscrowStale` block in `ReissueCredentials`, exactly as it was | **FAILED** — *"a re-issue marked a HEALTHY escrow stale…"* |
|
||||
| `TestEscrowStaleMechanism_StillWithholdsAndClears` | PASS | same mutation | **stayed GREEN** — correctly: the mutation restores a *caller*, not a break in the mechanism. That split is the evidence Scenario C's assertion is about the caller and not the flag. |
|
||||
| `TestEscrowPut_ChangedRepoKey_RaisesSignal` (D) | PASS | removed the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut` | **FAILED** — *"the repository key demonstrably changed and NO signal was raised"* |
|
||||
| `TestEscrowPut_UnchangedRepoKey_Silent`, `TestEscrowPut_HashlessSupersession_NoSignal` | PASS | same | stayed green — the detector's silence branches are independent |
|
||||
| controller `TestEscrowStale_MismatchWarnsOnceAndFlags` | PASS | — (cited as the continuous-coverage pin) | — |
|
||||
|
||||
**Nothing deleted on the storage endpoint**, including the ~1.2 GB of orphaned ciphertext — ruled,
|
||||
still owed, deliberately not ridden along with a drill.
|
||||
Scenario C asserts the **consequence** (the ACK still carries the hash, so auto-confirm can proceed)
|
||||
rather than the mechanism (that a function was not called), because the hash is what the drill's
|
||||
blockage actually turned on. It also asserts that `offsite_reissued` still fires — removing a false
|
||||
alarm must not remove the true notice.
|
||||
|
||||
## 8. The capability-map row
|
||||
## 7. Live validation
|
||||
|
||||
Now reads *"A customer's file survives a machine rebuild and comes back — the whole off-site story, end
|
||||
to end"*, **PROVEN-LIVE**. What it deliberately does **not** claim: the journey is manual and
|
||||
undocumented (R-204); the scope is `demo-hp` and a **controller-data** rebuild, **not** a total host
|
||||
loss and **not** a guest reprovision; and R-198's retention is still unit-proven.
|
||||
**Per task §12 point 5, a live re-issue was NOT run, and must not have been on demo-hp** — it would
|
||||
have been a credential rotation on the box holding the drill's evidence. Part 2 is proved by test and
|
||||
by the deployment being live and healthy. The controller-side halves of R-204 were validated live and
|
||||
are reported in `felhom-controller/REPORT.md`.
|
||||
|
||||
## 9. New findings — R-204, expanded into the gap list
|
||||
## 8. Register and documentation
|
||||
|
||||
The four steps between a recovered key and a restored file, all measured while walking them:
|
||||
- **`OPEN-ITEMS.md`** — **R-196 → CLOSED (hub v0.95.0)**; **R-204 → items 1–3 CLOSED, item 4 OPEN
|
||||
(→ R-193)** with its dependency named. The header block is updated and states explicitly that
|
||||
**R-202**, **the ~1.2 GB orphaned-ciphertext deletion** and **R-198's retention (still UNIT-PROVEN
|
||||
ONLY — the second deliberate wipe is the next item)** all **remain open**, so nothing is presumed
|
||||
closed by association. R-201 is recorded as PASSED. **R-204 is still the highest ID; nothing new
|
||||
was minted.**
|
||||
- **`architecture/00-capability-map.md`** — the recovery row now says three of the four crutches are
|
||||
gone, names the fixes and their evidence, and states that **item 4 (R-193) is the one that remains**
|
||||
and is why the row **keeps its "with a person present" qualifier**. The **R-199 back-pointer was
|
||||
already present** on the adjacent key-recovery row (added when that row was last corrected), so it
|
||||
needed no further action — verified, not assumed.
|
||||
- **`architecture/07-backup-architecture.md`** — **new §7.0, "What a customer can and cannot do
|
||||
ALONE"**: the four steps in a table with what each cost and its status, plus the honest current
|
||||
answer. This is the section a future reader will use to answer the question.
|
||||
- **`documentation/backlog/ROADMAP.md`** — R-196 and R-204 collapsed per the coupling rule.
|
||||
- **`CONTEXT.md`** — new standing ruling **S-32**, which supersedes S-31's steps 2–5 and carries the
|
||||
blinding mechanism, the fail-closed rule and the "no TTL" reasoning forward.
|
||||
- **`STATUS.md`** — rewritten to **one screen** (191 → ~90 lines) per its own header. It also had a
|
||||
corrupted, half-overwritten "What we're working on" section left from the drill session, which is
|
||||
now gone. Next item stated as the retention drill.
|
||||
|
||||
1. **R-193** — a rebuilt controller cannot configure its off-site tier (`no unconsumed offsite
|
||||
password`; the ledger shows its predecessor consumed it at 07:12:06). Needs an operator Re-issue.
|
||||
2. **The claim gate** — a rebuilt box is unclaimed, so *every* controller endpoint is intercepted. And
|
||||
**the local escape hatch does not work unaided**: `--print-reset-code` writes the new hash to
|
||||
`settings.json` while the running controller keeps its old copy in memory, so `effectiveClaimCode()`
|
||||
never sees it. **Restart the controller between minting and claiming** — two claim attempts failed
|
||||
before this was diagnosed.
|
||||
3. **R-196** — the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged, which
|
||||
gates every off-site run. Cleared with the **manual** confirm, never a ceremony.
|
||||
4. **`mode=unit` is the restore default** and returns the recovery unit, **not** the userdata leg. A
|
||||
customer told to "restore from off-site" gets their app definition and not their documents, and
|
||||
nothing in the outcome says so. **The worst of the four**, because it fails silently at the last
|
||||
step.
|
||||
**CI:** felhom.eu runs **154** (`5c7d671`, code) and **155** (`975a690`, manifest) — both success.
|
||||
`--no-verify` was **not** used; the pre-push gate ran and passed on every push.
|
||||
|
||||
## 10. CI
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
Docs push only; run number and task id in the session summary. **`--no-verify` not used.**
|
||||
|
||||
## 11. Observations — noticed, NOT acted on
|
||||
|
||||
1. **A same-day re-run replaces the day's snapshot** (`forget --keep-daily 7 --group-by host,tags`).
|
||||
Any drill depending on a specific snapshot surviving must account for it.
|
||||
2. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
|
||||
`kubectl port-forward`; the memory's `curl -u :$HUB_PW` recipe omits that.
|
||||
3. **`--print-reset-code` mints a new generation on every invocation** — three wasted generations here
|
||||
(4, 5, 6) while diagnosing the in-memory staleness.
|
||||
4. **The restore wrote into `backups/offsite-restore/<app>/` mirroring the full absolute path** — deep
|
||||
but unambiguous; worth knowing before writing customer-facing copy about where files land.
|
||||
- **`allowedEventTypes` still lists `escrow_stale`**, which after this change has **no producer** in
|
||||
either repo. It is inert rather than harmful; removing an allowlist entry is a behaviour change and
|
||||
is out of this session's scope.
|
||||
- `MarkEscrowStale` is now dead code by call-graph. Kept on purpose (§4 above) — but if a future
|
||||
session's linter or cleanup pass proposes deleting it, the reason it exists is in its doc comment
|
||||
and in the test that exercises it.
|
||||
- `/` on DooPlex is at **86%** used — under the 90% abort line, but worth watching before large builds.
|
||||
|
||||
Reference in New Issue
Block a user