docs: R-196 closed, R-204 items 1-3 closed, item 4 open (R-193)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
- OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and its dependency stated. Header restates that R-202, the 1.2 GB ciphertext deletion and R-198's still-unit-proven retention all REMAIN OPEN. - capability map: the recovery row keeps its 'with a person present' qualifier, names which crutch remains, and cites the three now gone. - 07-backup-architecture: new 7.0 - what a customer can and cannot do ALONE, the four steps in a table with status. This is the section a future reader will use to answer that question. - CONTEXT: standing ruling S-32, superseding S-31 steps 2-5. - STATUS: rewritten to one screen per its own header; removes a corrupted half-overwritten section left from the drill session. - ROADMAP: R-196 and R-204 collapsed.
This commit is contained in:
+39
@@ -17,6 +17,45 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-32 — THREE OF S-31's FOUR MANUAL STEPS ARE CLOSED (2026-08-05, R-204 items 1–3 / R-196).
|
||||
controller v0.198.0 + hub v0.95.0. Read this BEFORE S-31 — it supersedes S-31's steps 2–5.**
|
||||
|
||||
- **Step 1 (the reset code) — CLOSED.** `web.effectiveClaimCode` now calls `settings.ReloadClaimCode()`
|
||||
before applying the settings-vs-config precedence, so a code minted by the SEPARATE
|
||||
`--print-reset-code` process is visible without a restart. **The precedence rule is unchanged** —
|
||||
the defect was the freshness of the settings value, not which source wins. It is a READ-THROUGH on
|
||||
the claim path only, deliberately **not** a watcher, a signal handler or a TTL: a TTL would leave a
|
||||
window in which a superseded code still works, which is worse than the bug. `effectiveClaimCode`
|
||||
now returns an error and **every caller fails closed** (`claimGateActive` keeps the gate UP; an
|
||||
absent settings file is NOT an error — a pre-first-save box legitimately falls back to the
|
||||
controller.yaml bake). **S-31's "restart the controller between minting and claiming" is obsolete.**
|
||||
- **Steps 2–4 (the false staleness) — CLOSED.** `offsite.ReissueCredentials` no longer calls
|
||||
`MarkEscrowStale` and no longer emits `escrow_stale`; `offsite_reissued` is untouched. **The
|
||||
mechanism that made this expensive, recorded because it is not obvious:** `stale_at` makes
|
||||
`GetEscrowStatusForCustomer` WITHHOLD `restic_pw_sha256` — which is the exact value the controller's
|
||||
Scenario-F re-check (`report.EscrowAutoConfirmer.reconcileEscrowed`) compares against its live repo
|
||||
password on every ACK. **The precautionary mark was blinding the measurement that would have told
|
||||
the truth**, leaving the box able to report only the hash-LESS reason, which is false. Coverage
|
||||
after removal: that per-ACK comparison (continuous, box-side) plus R-197's `offsite_repo_key_changed`
|
||||
(edge-triggered, hub-side, at a supersession). **`MarkEscrowStale` is KEPT with no caller** — the
|
||||
flag is still read by the ACK, the operator card and the PBS-DR view, and the right way to set it is
|
||||
a future caller that has MEASURED a change; `TestEscrowStaleMechanism_StillWithholdsAndClears` stops
|
||||
it decaying meanwhile. **The manual `/backup/offbox/confirm-escrow` workaround is no longer needed
|
||||
after a Re-issue.**
|
||||
- **Step 5 (the restore default) — CLOSED.** `restoreScratchOutcomeMsg` states, for `mode=unit`, what
|
||||
came back, that the customer's own files did NOT, and the step that gets them; the wizard's intent
|
||||
card 1 states its scope before the choice. **The `mode=full` size gate is untouched** and pinned as
|
||||
unchanged. **The default stays `unit`:** all three wizard forms set `mode` explicitly, so changing
|
||||
it would alter nothing the customer sees while silently changing a hand-crafted mode-less POST.
|
||||
- **WHAT REMAINS is S-31's step 1-before-the-list: a rebuilt box still cannot obtain an off-site
|
||||
credential unaided** (the one-time password was consumed by its predecessor), so an operator
|
||||
Re-issue is still required. That is **R-193**, it needs an operator ruling, and it is why the
|
||||
capability-map recovery row keeps its *"with a person present"* qualifier.
|
||||
- **Live-validated on demo boxes, endpoint-level** (no browser on DooPlex): reset codes on
|
||||
demo-felhom 9201 with `restarts=0` across the mints; the unit-restore outcome and the full-restore
|
||||
size-gate reveal on demo-hp 9201, using `privatebin` so the drill's `calibre-web` scratch was not
|
||||
touched. **The Part 2 change was NOT fired live on demo-hp** — a Re-issue there was out of scope.
|
||||
|
||||
**S-31 — THE DRILL PASSED: a customer's file survives a machine rebuild and comes back byte-identical.
|
||||
The capability is proven; the customer JOURNEY is four undocumented manual steps (2026-08-04, R-201/R-204).**
|
||||
|
||||
|
||||
@@ -1,124 +1,160 @@
|
||||
# REPORT — R-201: **THE DRILL PASSED.** A customer's file survived a machine rebuild
|
||||
# REPORT — R-196 / R-204 item 2 (hub v0.95.0), 2026-08-05
|
||||
|
||||
**2026-08-04, 21:30–23:15** · `demo-hp` deliberately rebuilt · no code, no version bump ·
|
||||
`demo-felhom` untouched. Record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.
|
||||
**A re-issue no longer marks a healthy escrow stale.** One behaviour change, one register closed, and
|
||||
the coverage claim proved rather than assumed. The controller's half of R-204 (items 1 and 3) is
|
||||
`felhom-controller` v0.198.0.
|
||||
|
||||
---
|
||||
## 1. Baselines, re-read on arrival
|
||||
|
||||
## 1. THE VERDICT — PASS
|
||||
| Repo | Expected | Found |
|
||||
|---|---|---|
|
||||
| `felhom.eu` | `2a7ac03c4726` / hub v0.94.0 (deployed `felhom-hub:0.94.0`) | **exact match**, tree clean, `HEAD == origin/main` |
|
||||
|
||||
```
|
||||
pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||
restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||
```
|
||||
**§3.2's landmark had DRIFTED, and the drift changed the work.** The task described
|
||||
`offsite.go:222-231` under a known-consequence comment saying the mark was made on a false premise.
|
||||
That comment had already been rewritten by the R-196 comment-correction commit, and the version on
|
||||
`main` gave a **non-false** ground for the mark: *"the box's re-apply may mint a fresh repository
|
||||
password (it does exactly that whenever `<DataDir>/offbox/repo_password` is absent — the
|
||||
guest-rebuild shape)"*. So the question was no longer "delete a comment's lie" but "is the shape it
|
||||
guards actually covered elsewhere?" — which is Scenario D, and §8.2 says to stop and report if it is
|
||||
not. It is; §3 below is the evidence.
|
||||
|
||||
**Byte-identical.** The controller's data volume was destroyed and the file deleted from disk; the
|
||||
customer's recovery code, the customer's restore flow, and the file came back. **The Felhom backup
|
||||
story is proved end to end for the first time.**
|
||||
## 2. What changed
|
||||
|
||||
## 2. Snapshot count at step 9 — **3, not 1**
|
||||
`offsite.ReissueCredentials` no longer calls `MarkEscrowStale` and no longer emits the `escrow_stale`
|
||||
event. **`offsite_reissued` is untouched** and still fires on every re-issue. The known-consequence
|
||||
comment is rewritten to record what was done, when, and why — with the disagreement below stated in
|
||||
it rather than absorbed.
|
||||
|
||||
`repo_state: null` (the repository **opened**, not orphaned), `snapshot_count: 3`, `repo_size_bytes:
|
||||
42 026` — the pre-wipe size exactly. The pre-wipe snapshot `e6132ae5` was still present with the file
|
||||
inside it. **No fresh history was started over the old one** — the failure this drill was designed to
|
||||
catch did not occur.
|
||||
**What the mark actually cost, established mechanically rather than asserted** (this is why it was a
|
||||
blocker and not a nit):
|
||||
|
||||
## 3. §5's five conditions, recorded before the wipe
|
||||
1. `stale_at` set → `GetEscrowStatusForCustomer` **withholds** `restic_pw_sha256` from the report ACK.
|
||||
2. With no hash, the controller's SLICE-3 auto-confirm returns early and cannot flip
|
||||
`pending → escrowed`.
|
||||
3. `OffboxRunnable() = OffboxConfigured() && EscrowState == "escrowed"` → **every off-site backup
|
||||
refused**, indefinitely, on a box whose repository key was never in doubt.
|
||||
4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
|
||||
supersedes the sealed blob. **During a recovery that is the one act that would have destroyed the
|
||||
key just recovered.**
|
||||
|
||||
1. sentinel **listed by name** in snapshot `e6132ae5` (19:36:26), 181 B.
|
||||
2. rollback archive `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` (1 606 765 083 B) **verified by a
|
||||
full zstd stream read** (4 867 573 760 B) with the sentinel confirmed inside.
|
||||
3. §3's option — §4 below.
|
||||
4. `demo-felhom` `health=ok`, `escrow_state=escrowed`, untouched throughout.
|
||||
5. nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.
|
||||
## 3. Scenario D — the evidence that the removed marking is covered
|
||||
|
||||
**One precondition had drifted and was repaired, not worked around:** the staged snapshot had lost the
|
||||
sentinel because the afternoon's experiment produced a *later* same-day snapshot and `forget
|
||||
--keep-daily 7 --group-by host,tags` pruned the good one. One backup re-established it, re-verified by
|
||||
listing. **A good snapshot is not durable against a later bad run on the same day.**
|
||||
The mark was precautionary and aimed at ONE shape: a re-issue followed by a box that mints a fresh
|
||||
repository password (the guest-rebuild shape, where `offbox/repo_password` is absent). That shape is
|
||||
measured in two independent places, and **the mark was blinding one of them**:
|
||||
|
||||
## 4. §3 — the recovery code
|
||||
- **Continuous, box-side — the real coverage.** `report.EscrowAutoConfirmer.reconcileEscrowed`
|
||||
(controller) compares the ACK's sealed `restic_pw_sha256` against the box's CURRENT local repo
|
||||
password on **every report ACK**, raising the stale flag, the customer card and the
|
||||
„create a new recovery code" CTA on a mismatch. That is a **measurement**, not a guess, and it is
|
||||
continuous rather than edge-triggered. Pinned by the controller's
|
||||
`TestEscrowStale_MismatchWarnsOnceAndFlags` — re-run green this session.
|
||||
**And step 1 above was blinding it:** a stale flag empties the very hash that comparison needs, so
|
||||
the box could only reach the hash-LESS branch and report *"the hub's current blob carries NO
|
||||
password hash"* — which is false. Removing the mark restores the true signal.
|
||||
- **Edge-triggered, hub-side.** R-197's `offsite_repo_key_changed` fires from
|
||||
`api.maybeEmitRepoKeyChanged` on a proven hash difference across a supersession and pages the
|
||||
operator. **Red-proved:** removing the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut`
|
||||
made `TestEscrowPut_ChangedRepoKey_RaisesSignal` fail with *"the repository key demonstrably changed
|
||||
and NO signal was raised"*, while the two silence tests stayed green.
|
||||
|
||||
**Option B as already in place, improved: no new copy was made, so nothing needed shredding.** The
|
||||
operator's own `~/.config/credentials` (`0600`) was the source; the code was **piped to stdin** for
|
||||
each invocation — never an argument, never exported, never a second file, never logged. Destroying the
|
||||
operator's own store would have destroyed their record; because no extra copy existed, there is nothing
|
||||
left to prove gone.
|
||||
**Disagreement recorded, per the R-96 standing rule.** Scenario D as written asks that a real key
|
||||
change also *"mark the escrow stale"*. **It must not, and nothing was changed to make it.** The hub
|
||||
learns of a real change at the instant a supersession **seals the new password** — i.e. when the
|
||||
escrow is at its freshest. Marking it stale there would ask the customer for a ceremony to fix the
|
||||
ceremony that just ran. The correct consequence at that instant is the operator alarm, which is
|
||||
exactly what R-197 already does. This is recorded in the code comment, the CHANGELOG and OPEN-ITEMS,
|
||||
not only here.
|
||||
|
||||
**Verified anyway, with the planted-copy positive control:** 0 hits in the agent journal, 0 in the
|
||||
controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover
|
||||
`felhom-idesc-*` dirs — and the same sweep found a planted copy (**1**), then **0** after shredding it.
|
||||
## 4. `MarkEscrowStale` is kept with no caller — deliberately
|
||||
|
||||
## 5. Every step's observable
|
||||
Per task §5 it was not to be modified, and it is not deleted either. The `stale_at` flag remains live
|
||||
and correct — read by the ACK, the operator config card and the PBS-DR view — and the right way to
|
||||
set it is a **future EVIDENTIAL caller** that has measured a key change rather than guessed at one.
|
||||
Its doc comment now says so plainly instead of naming a caller that no longer exists, and
|
||||
`TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying to inert while
|
||||
nothing writes it (the seam-built-but-never-wired shape, in reverse).
|
||||
|
||||
| step | observable |
|
||||
The schema comment and `EscrowStatus.Stale`'s comment were corrected the same way — each of the three
|
||||
previously asserted a writer that is now gone.
|
||||
|
||||
## 5. Files modified
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| 6 | fresh data dir `20:00:2x`; new `encryption.key`; `claimed = None`; `offbox = null` |
|
||||
| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED.** Nothing re-escrowed itself |
|
||||
| **8** | **recovered `8a9e33aa4da6…`** — matches the pre-wipe on-disk key and the hub's record |
|
||||
| 9a | `[INSTALLED] … reads back identical` |
|
||||
| 9b | after Re-issue + apply, the on-disk key is **still** the recovered one |
|
||||
| 9c | **repository OPENED** — `repo_state: null`, 3 snapshots, 42 026 B |
|
||||
| **10** | **restored sha256 byte-identical** |
|
||||
| 11 | this record |
|
||||
| `hub/internal/offsite/offsite.go` | the pessimistic `MarkEscrowStale` + `escrow_stale` event removed; comment rewritten to record the change, the coverage and the disagreement |
|
||||
| `hub/internal/offsite/offsite_test.go` | `TestReissue_InvalidatesEscrow` **replaced in place by its exact inverse** `TestReissue_DoesNotMarkAHealthyEscrowStale`; new `TestEscrowStaleMechanism_StillWithholdsAndClears` |
|
||||
| `hub/internal/store/store.go` | three comments corrected (`MarkEscrowStale`, the `stale_at` schema note, `EscrowStatus.Stale`) — each had named a writer that no longer exists |
|
||||
| `manifests/hub.yaml` | image tag `0.94.0` → `0.95.0` |
|
||||
| `hub/CHANGELOG.md`, `CONTEXT.md`, `STATUS.md`, `documentation/…` | v0.95.0 entry; ruling **S-32**; the register and architecture updates below |
|
||||
|
||||
Superseded escrow rows remained **2** throughout — **no ceremony was run at any point.**
|
||||
**Commits on `main`:** `d1a8edb` (behaviour + tests + comments) · `5c7d671` (CHANGELOG) ·
|
||||
`975a690` (manifest bump).
|
||||
|
||||
## 6. Part 2 — not run
|
||||
**Deploy:** built + pushed `felhom-hub:0.95.0`, bumped `manifests/hub.yaml`, pushed, then a
|
||||
**deliberate ArgoCD hard-refresh + sync** (auto-sync stays off; no `kubectl set image` anywhere).
|
||||
Result: app `felhom` **Synced / Healthy**, `deploy/hub` rolled out, running image
|
||||
`gitea.dooplex.hu/admin/felhom-hub:0.95.0`, startup log clean (offsite provisioning, pool-box checker,
|
||||
PBS-DR reconciler and all six host checkers initialised; `Listening on :8080`).
|
||||
|
||||
Gate: *"the drill PASSED; its evidence is written down; and there is time."* The first two are met; the
|
||||
third is not — 23:15, after a full destructive cycle, and Part 2 is a second wipe. Deliberately left
|
||||
for a fresh session with drill 1's result already recorded, so a second wipe cannot overwrite it.
|
||||
**R-198's retention remains unit-proven only** — nothing has yet superseded a key in production, and it
|
||||
is now the last unproven link in this chain.
|
||||
## 6. Tests and red-proofs
|
||||
|
||||
## 7. Teardown — three layers
|
||||
Green gate: `cd hub && go build ./... && go vet ./... && go test ./...` — **rc=0**.
|
||||
`python3 scripts/repo_gates.py --fast` — **all five gates OK**.
|
||||
|
||||
| layer | state |
|
||||
|---|---|
|
||||
| the guest | **healthy and re-armed**: controller v0.197.0 claimed, six app containers serving, all three apps re-toggled for off-site, the sentinel restored to its live location and verified. The verification-copy tree is **kept as evidence**. Scratch band empty |
|
||||
| the host | vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted |
|
||||
| the hub | **no new customer records** — `demo-hp` is the same row throughout (a controller-data rebuild, not a re-enrolment). One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. **The two superseded escrow rows are unchanged** |
|
||||
| Test | Result | Red-proof — what was mutated | Outcome |
|
||||
|---|---|---|---|
|
||||
| `TestReissue_DoesNotMarkAHealthyEscrowStale` (C) | PASS | restored the pessimistic `MarkEscrowStale` block in `ReissueCredentials`, exactly as it was | **FAILED** — *"a re-issue marked a HEALTHY escrow stale…"* |
|
||||
| `TestEscrowStaleMechanism_StillWithholdsAndClears` | PASS | same mutation | **stayed GREEN** — correctly: the mutation restores a *caller*, not a break in the mechanism. That split is the evidence Scenario C's assertion is about the caller and not the flag. |
|
||||
| `TestEscrowPut_ChangedRepoKey_RaisesSignal` (D) | PASS | removed the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut` | **FAILED** — *"the repository key demonstrably changed and NO signal was raised"* |
|
||||
| `TestEscrowPut_UnchangedRepoKey_Silent`, `TestEscrowPut_HashlessSupersession_NoSignal` | PASS | same | stayed green — the detector's silence branches are independent |
|
||||
| controller `TestEscrowStale_MismatchWarnsOnceAndFlags` | PASS | — (cited as the continuous-coverage pin) | — |
|
||||
|
||||
**Nothing deleted on the storage endpoint**, including the ~1.2 GB of orphaned ciphertext — ruled,
|
||||
still owed, deliberately not ridden along with a drill.
|
||||
Scenario C asserts the **consequence** (the ACK still carries the hash, so auto-confirm can proceed)
|
||||
rather than the mechanism (that a function was not called), because the hash is what the drill's
|
||||
blockage actually turned on. It also asserts that `offsite_reissued` still fires — removing a false
|
||||
alarm must not remove the true notice.
|
||||
|
||||
## 8. The capability-map row
|
||||
## 7. Live validation
|
||||
|
||||
Now reads *"A customer's file survives a machine rebuild and comes back — the whole off-site story, end
|
||||
to end"*, **PROVEN-LIVE**. What it deliberately does **not** claim: the journey is manual and
|
||||
undocumented (R-204); the scope is `demo-hp` and a **controller-data** rebuild, **not** a total host
|
||||
loss and **not** a guest reprovision; and R-198's retention is still unit-proven.
|
||||
**Per task §12 point 5, a live re-issue was NOT run, and must not have been on demo-hp** — it would
|
||||
have been a credential rotation on the box holding the drill's evidence. Part 2 is proved by test and
|
||||
by the deployment being live and healthy. The controller-side halves of R-204 were validated live and
|
||||
are reported in `felhom-controller/REPORT.md`.
|
||||
|
||||
## 9. New findings — R-204, expanded into the gap list
|
||||
## 8. Register and documentation
|
||||
|
||||
The four steps between a recovered key and a restored file, all measured while walking them:
|
||||
- **`OPEN-ITEMS.md`** — **R-196 → CLOSED (hub v0.95.0)**; **R-204 → items 1–3 CLOSED, item 4 OPEN
|
||||
(→ R-193)** with its dependency named. The header block is updated and states explicitly that
|
||||
**R-202**, **the ~1.2 GB orphaned-ciphertext deletion** and **R-198's retention (still UNIT-PROVEN
|
||||
ONLY — the second deliberate wipe is the next item)** all **remain open**, so nothing is presumed
|
||||
closed by association. R-201 is recorded as PASSED. **R-204 is still the highest ID; nothing new
|
||||
was minted.**
|
||||
- **`architecture/00-capability-map.md`** — the recovery row now says three of the four crutches are
|
||||
gone, names the fixes and their evidence, and states that **item 4 (R-193) is the one that remains**
|
||||
and is why the row **keeps its "with a person present" qualifier**. The **R-199 back-pointer was
|
||||
already present** on the adjacent key-recovery row (added when that row was last corrected), so it
|
||||
needed no further action — verified, not assumed.
|
||||
- **`architecture/07-backup-architecture.md`** — **new §7.0, "What a customer can and cannot do
|
||||
ALONE"**: the four steps in a table with what each cost and its status, plus the honest current
|
||||
answer. This is the section a future reader will use to answer the question.
|
||||
- **`documentation/backlog/ROADMAP.md`** — R-196 and R-204 collapsed per the coupling rule.
|
||||
- **`CONTEXT.md`** — new standing ruling **S-32**, which supersedes S-31's steps 2–5 and carries the
|
||||
blinding mechanism, the fail-closed rule and the "no TTL" reasoning forward.
|
||||
- **`STATUS.md`** — rewritten to **one screen** (191 → ~90 lines) per its own header. It also had a
|
||||
corrupted, half-overwritten "What we're working on" section left from the drill session, which is
|
||||
now gone. Next item stated as the retention drill.
|
||||
|
||||
1. **R-193** — a rebuilt controller cannot configure its off-site tier (`no unconsumed offsite
|
||||
password`; the ledger shows its predecessor consumed it at 07:12:06). Needs an operator Re-issue.
|
||||
2. **The claim gate** — a rebuilt box is unclaimed, so *every* controller endpoint is intercepted. And
|
||||
**the local escape hatch does not work unaided**: `--print-reset-code` writes the new hash to
|
||||
`settings.json` while the running controller keeps its old copy in memory, so `effectiveClaimCode()`
|
||||
never sees it. **Restart the controller between minting and claiming** — two claim attempts failed
|
||||
before this was diagnosed.
|
||||
3. **R-196** — the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged, which
|
||||
gates every off-site run. Cleared with the **manual** confirm, never a ceremony.
|
||||
4. **`mode=unit` is the restore default** and returns the recovery unit, **not** the userdata leg. A
|
||||
customer told to "restore from off-site" gets their app definition and not their documents, and
|
||||
nothing in the outcome says so. **The worst of the four**, because it fails silently at the last
|
||||
step.
|
||||
**CI:** felhom.eu runs **154** (`5c7d671`, code) and **155** (`975a690`, manifest) — both success.
|
||||
`--no-verify` was **not** used; the pre-push gate ran and passed on every push.
|
||||
|
||||
## 10. CI
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
Docs push only; run number and task id in the session summary. **`--no-verify` not used.**
|
||||
|
||||
## 11. Observations — noticed, NOT acted on
|
||||
|
||||
1. **A same-day re-run replaces the day's snapshot** (`forget --keep-daily 7 --group-by host,tags`).
|
||||
Any drill depending on a specific snapshot surviving must account for it.
|
||||
2. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
|
||||
`kubectl port-forward`; the memory's `curl -u :$HUB_PW` recipe omits that.
|
||||
3. **`--print-reset-code` mints a new generation on every invocation** — three wasted generations here
|
||||
(4, 5, 6) while diagnosing the in-memory staleness.
|
||||
4. **The restore wrote into `backups/offsite-restore/<app>/` mirroring the full absolute path** — deep
|
||||
but unambiguous; worth knowing before writing customer-facing copy about where files land.
|
||||
- **`allowedEventTypes` still lists `escrow_stale`**, which after this change has **no producer** in
|
||||
either repo. It is inert rather than harmful; removing an allowlist entry is a behaviour change and
|
||||
is out of this session's scope.
|
||||
- `MarkEscrowStale` is now dead code by call-graph. Kept on purpose (§4 above) — but if a future
|
||||
session's linter or cleanup pass proposes deleting it, the reason it exists is in its doc comment
|
||||
and in the test that exercises it.
|
||||
- `/` on DooPlex is at **86%** used — under the 90% abort line, but worth watching before large builds.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Updated 2026-08-04.**
|
||||
**Updated 2026-08-05.**
|
||||
|
||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
||||
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
||||
@@ -13,179 +13,78 @@
|
||||
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
||||
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
||||
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
||||
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
||||
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
|
||||
box six times, everything returned every time, and an app switched off deliberately stayed off.
|
||||
Proven end to end on real hardware.
|
||||
places — the machine's drive, a second drive, and an encrypted off-site copy.
|
||||
|
||||
**And the whole backup promise is now proved.** On 4 August we destroyed a machine on purpose and
|
||||
deleted a marked file from its disk. Using the recovery code you saved: the backup key came back
|
||||
**identical, character for character**; the existing off-site store **opened** rather than starting
|
||||
over; and the file was restored **byte for byte identical**. *(R-201)*
|
||||
|
||||
## What's broken
|
||||
|
||||
- **A customer still cannot do that recovery alone — but only one step is left.** Getting from "the
|
||||
key is recoverable" to "the file is back" took four steps that appeared in no instructions. **Three
|
||||
are fixed today** *(R-204)*: the local reset-code tool works on the first try instead of needing the
|
||||
controller restarted; re-issuing the storage credential no longer falsely marks the recovery key
|
||||
"stale" (which used to stop every off-site backup and invite the one act that would have destroyed
|
||||
the recovered key); and the everyday restore now **says in plain Hungarian that it returned the
|
||||
app's settings and database and not your documents**, and names the button that does. **The step
|
||||
that remains is the first one:** a rebuilt machine cannot get a storage credential by itself,
|
||||
because the one-time password was used up by its predecessor — so you still have to press
|
||||
Re-issue. That is a design decision waiting on you, below. *(R-193)*
|
||||
- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
|
||||
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one, so the old
|
||||
backups can no longer be opened by the machine that wrote them. Both demo machines did this on 3–4
|
||||
August — demo-hp lost access to 15 backups (41 MB), demo-felhom to 36 (1.14 GB), and **the second one
|
||||
happened in silence for thirteen hours**. From today that silence is fixed (below) and the old key
|
||||
survives the ceremony (below), but **the rebuild itself still starts a fresh history**, and getting
|
||||
the old one back is a path nobody has walked. *(R-193)*
|
||||
- *(fixed 4 Aug)* ~~The safety net we believed was under all of this is not there.~~ **The central hub
|
||||
was keeping the wrong key, and the fix shipped the same day.** When a machine re-seals its backups
|
||||
under a new recovery code, the hub keeps the old sealed package — but it was keeping the one for the
|
||||
whole-machine backups and **not** the one for the off-site file backups, which is the whole point. And
|
||||
because the swap happens in one motion, **the act that destroyed the old key was the recovery-code
|
||||
ceremony itself** — the exact thing a rebuilt machine asks its customer to do, on a screen promising
|
||||
the old backups stay recoverable. Both keys are now kept. **What this cannot undo:** anything
|
||||
superseded before today is gone for good, which includes both demo machines' pre-4-August keys — so
|
||||
those 51 orphaned backups were beyond reach even if the recovery codes had been kept. *(R-198)*
|
||||
- **IT WORKS. A customer's file survived a machine being destroyed and came back — proved last night on
|
||||
real hardware, end to end.** We wiped the HP machine's controller data on purpose and deleted the
|
||||
marked file from its disk. Using the recovery code you saved: the backup key came back **identical,
|
||||
character for character**; the existing off-site store **opened** rather than starting over (the same
|
||||
three backups, the same 42 026 bytes as before the wipe); and the file was restored **byte for byte
|
||||
identical**. That is the whole backup promise, demonstrated for the first time. *(R-201)*
|
||||
- **But a customer could not have done it alone, and that is now the open work.** Getting from "the key
|
||||
is recoverable" to "the file is back" took four steps that appear in no instructions: re-issuing the
|
||||
storage credential; re-claiming the machine (whose local reset-code tool **does not work until the
|
||||
controller is restarted** — two attempts failed before we spotted it); manually clearing a "stale"
|
||||
mark that the re-issue sets even though the key never changed; and choosing the *full* restore,
|
||||
because the default one returns the app's settings and **not** the customer's documents — with
|
||||
nothing saying so. **The capability is real; the experience is not built yet.** *(R-204)*
|
||||
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one. Both demo
|
||||
machines did this on 3–4 August — 51 backups (~1.2 GB) between them. The old key now survives the
|
||||
recovery ceremony, and a changed key now raises an alarm the same day, but **the rebuild itself
|
||||
still starts a fresh history**. *(R-193)*
|
||||
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
|
||||
says the old backups may later be restorable with the matching recovery code. From today that is true
|
||||
for machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||||
cannot tell which case it is in without the recovery path above. We deliberately did **not** patch the
|
||||
sentence: a conditional promise that can still be wrong is worse on that screen than a vague one.
|
||||
*(R-202)*
|
||||
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
|
||||
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
|
||||
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
|
||||
it every day since 27 July. Both machines and all future ones. *(R-191)*
|
||||
says the old backups may later be restorable with the matching recovery code. That is true for
|
||||
machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||||
cannot tell which case it is in. We deliberately did **not** patch the sentence: a conditional
|
||||
promise that can still be wrong is worse there than a vague one. *(R-202)*
|
||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap.
|
||||
*(R-95, R-87)*
|
||||
also delete it. A daily snapshot is armed as a stopgap. *(R-95, R-87)*
|
||||
|
||||
## What shipped recently
|
||||
|
||||
- **The daily error email about David is gone.** David's record was created when he said he would try
|
||||
Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every
|
||||
morning at five the system reported that a database backup had not happened. **A customer with no
|
||||
machine ever attached now produces no alarm.** A machine that *was* attached and went quiet still
|
||||
does, because that is a real fault, and there is a test whose whole job is to make sure a future
|
||||
change cannot take that away. David's record was not touched — the record was right; the alarm was
|
||||
wrong. **The first quiet morning will be 5 August, so this is not confirmed yet.** *(R-195)*
|
||||
|
||||
- **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we
|
||||
found a permission that vanished with no reinstall and no recorded action; the machine now notices,
|
||||
restores it through the same controlled helper that grants it in the first place, and **raises an
|
||||
alert saying it was missing and was restored**. The alert is the point: a permission that silently
|
||||
fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo
|
||||
machine by taking the permission away by hand. *(R-190 — the repair; why it vanished is still open)*
|
||||
|
||||
- **The on-machine backup copy has now been proved to restore — by the machines themselves**, both of
|
||||
them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the
|
||||
off-site copy. *(R-86, R-185)*
|
||||
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
||||
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
||||
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
||||
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
|
||||
written to disk with the result and survives. This was caught happening, not predicted: a real
|
||||
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
|
||||
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
|
||||
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
||||
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
||||
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
||||
|
||||
- **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time;
|
||||
a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
|
||||
restore no longer quietly returns the wrong thing. *(R-204 items 1–3, R-196)*
|
||||
- **2026-08-04 (night)** — **The drill PASSED**, end to end, on real hardware. *(R-201)*
|
||||
- **2026-08-04** — The folder-left-out-of-the-backup problem fixed both halves: the app and its backup
|
||||
look in the same directory, and a backup that misses a folder marked essential reports *incomplete*
|
||||
instead of success. *(R-203)*
|
||||
- **2026-08-04** — The hub now keeps the off-site backup key when a machine re-seals, instead of only
|
||||
the whole-machine one, and a machine can fetch its own sealed package back. *(R-198, R-199)*
|
||||
- **2026-08-04** — The daily false alarm about David is gone; a vanished permission now repairs itself
|
||||
**and says it had to**; the weekly off-site backup stopped reporting failure after a successful
|
||||
upload. *(R-195, R-190, R-191)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** making that journey something a customer can actually follow — the four steps above, worst
|
||||
first: the restore default that silently returns the wrong thing. The HP machine is sitting
|
||||
mid-drill with the recovered key already on it; it needs re-claiming and one setting confirmed, then
|
||||
the backup runs and the file is restored. **Do not let it make a new recovery code** — that would
|
||||
The marked file now lands in
|
||||
the off-site backup, so there is finally something to recover. Everything else is already in place on
|
||||
the HP machine — the recovery code you saved, a working off-site store, the app and the file. Nothing
|
||||
has been wiped; that step waits for your go-ahead at a marked stop point. Both honesty fixes shipped today: a changed backup key now
|
||||
raises an alarm on the day, and the email that stated the opposite of what it measured now describes
|
||||
what it actually saw.
|
||||
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
|
||||
refuse and ask, rather than start fresh — but that is a prediction from the code, not something
|
||||
anyone has watched happen.
|
||||
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
||||
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
||||
*(R-87)*
|
||||
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
||||
- **Next: the retention proof.** The hub keeping the old sealed key when a machine re-seals is the one
|
||||
remaining link that has **never run outside a test**. Proving it needs a second deliberate wipe on
|
||||
the spare demo machine, and it is its own procedure. *(R-198)*
|
||||
- **Then:** the orphaned-backup deletion you asked for (below), and the off-site copy the machine can
|
||||
still erase. *(R-193, R-95)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- *(decided 4 Aug — recorded here until the work it unblocks is done)* You chose **not** to keep a
|
||||
copy of the backup key on the Proxmox host. That is a coherent choice and it has a consequence worth
|
||||
restating: **the customer's own recovery code is now the only route back from a rebuild**, so the
|
||||
sealed package the hub keeps is the single copy. Today's fix is what makes that true rather than
|
||||
wishful — until this morning the ceremony destroyed it. *(R-193)*
|
||||
- **Your ruling about the recovery screen has been priced, and it can be built.** You said: a freshly
|
||||
installed machine that finds a sealed package waiting should say so loudly, offer the customer a box
|
||||
to type their recovery code into, and show what would come back before doing anything. **All of it is
|
||||
buildable, and one part is already free** — the hub is *already* telling every machine, on every
|
||||
check-in, that a sealed package exists and which key it covers, and the machine currently throws that
|
||||
message away. Showing a preview is also cheap: listing what is in an off-site store reads it without
|
||||
writing to it, so the customer can see how many backups, from when, and for which apps before
|
||||
committing. The real work is one new connection: the unsealing has to happen in the part that runs on
|
||||
the Proxmox host, because the customer-facing part cannot do it — the same crossing your recovery-code
|
||||
ceremony already makes in the other direction. **One thing for you to weigh, which we deliberately did
|
||||
not decide:** a screen that takes a recovery code and then shows what is in a backup store is reachable
|
||||
by anyone with the household's dashboard password, and the preview reveals backup dates and app names.
|
||||
*(R-193)*
|
||||
- **Should we run the proof?** Nothing about recovery has ever actually been done, so we designed the
|
||||
exercise: on the spare demo machine, make a recovery code and **keep it**, put a marked file into an
|
||||
app, back it up off-site, wipe the machine, reinstall, recover with the saved code, and check the
|
||||
marked file comes back byte for byte. About half a day, of which under an hour needs you. The failure
|
||||
we are watching for is precise: **if the recovered machine reports one backup instead of the ones we
|
||||
put there, it started a fresh history and the proof failed** — "the store opened" is not good enough.
|
||||
It costs that machine's contents if it goes wrong, which is acceptable there. **You said yes, after
|
||||
R-198 — and R-198 shipped this morning, so this is ready to run on your word.** *(R-201)*
|
||||
- **The one-shot credential decision — this is now the last thing between a customer and an unaided
|
||||
recovery.** A rebuilt machine has no storage credential of its own, so an operator must press
|
||||
Re-issue. Everything after that point is now self-service. Deciding how a rebuilt machine should get
|
||||
a credential is the remaining design question. *(R-193, R-204 item 4)*
|
||||
- **The recovery screen you described has been priced, and it can be built.** A freshly installed
|
||||
machine that finds a sealed package waiting should say so, offer a box for the recovery code, and
|
||||
show what would come back before doing anything. One thing to weigh, deliberately not decided: that
|
||||
screen is reachable by anyone with the household's dashboard password, and the preview reveals
|
||||
backup dates and app names. *(R-193)*
|
||||
- **The orphaned backups on the storage box — you said delete, and it is still owed.** About 1.2 GB
|
||||
across the two demo machines, in set-aside stores nobody can open and nothing prunes. Deliberately not
|
||||
done today: erasing data on the storage endpoint is not something that should ride along with a
|
||||
database change, so it wants its own session. *(R-193)*
|
||||
across the two demo machines, in set-aside stores nobody can open and nothing prunes. It wants its
|
||||
own session rather than riding along with other work. *(R-193)*
|
||||
- *(decided 4 Aug)* You chose **not** to keep a copy of the backup key on the Proxmox host, which
|
||||
makes the customer's own recovery code the only route back from a rebuild. *(R-193)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
machines to install, only which ones exist. Closing that needs either a password given to the build
|
||||
server or a check inside the hub itself. *(R-184)*
|
||||
- **Nothing else.**
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-04 (night)** — **The drill PASSED.** We destroyed a machine on purpose and the customer's
|
||||
file came back byte-for-byte identical, using the recovery code you saved. First time the backup
|
||||
story has been proved end to end. It needed four undocumented manual steps to get there, which are
|
||||
now the next piece of work. The machine is up, healthy and re-armed. *(R-201, R-204)*
|
||||
- **2026-08-04 (evening)** — **Fixed the folder-left-out-of-the-backup problem, both halves.** The app
|
||||
and its backup now look in the same directory, and a backup that misses a folder marked essential
|
||||
reports *incomplete* instead of success. Proved by listing the backup's own contents and finding the
|
||||
marked file. The wipe-and-restore proof is unblocked. *(R-203, R-201)*
|
||||
- **2026-08-04 (late)** — Set up the wipe-and-restore proof on the HP machine and **stopped before the
|
||||
wipe**: a folder marked as protected was missing from the off-site backup while the backup reported
|
||||
success. Three things were proved on the way, all firsts: a rebuilt machine's off-site backup now
|
||||
**refuses and says so** instead of quietly starting over; the "start a new store" repair works and
|
||||
keeps the old data aside; and the HP machine's pre-3-August history is gone for good — its key was
|
||||
destroyed four hours before the fix that would have kept it. *(R-203, R-201, R-193)*
|
||||
- **2026-08-04 (evening)** — **Proved the backup key comes back.** A demo machine fetched its own
|
||||
sealed package, opened it with the saved recovery code, and produced a key identical to the one it
|
||||
uses. A wrong code was refused and wrote nothing. The recovery code left no trace anywhere on the
|
||||
machine — searched, with a planted copy first to prove the search worked. *(R-199, R-200)*
|
||||
- **2026-08-04 (earlier)** — **Fixed the worst of it.** The hub now keeps the off-site backup key when a
|
||||
machine re-seals, instead of only the whole-machine one — four lines of database change that stop the
|
||||
recovery ceremony from destroying the thing it exists to protect. A changed backup key now raises an
|
||||
alarm the same day. The daily email that described the opposite of what it measured now states what it
|
||||
saw, and a refusal to auto-repair leaves a record instead of silence. *(R-198, R-197, R-192, R-196)*
|
||||
- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
|
||||
password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
|
||||
access to their off-site history, one of them silently. A decision is now waiting on you. The daily
|
||||
false alarm about David is fixed. *(R-193, R-195)*
|
||||
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
||||
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
||||
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
||||
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
||||
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
||||
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|
||||
- **One small question, not urgent.** The automatic version check cannot see which version you have
|
||||
told machines to install, only which ones exist. *(R-184)*
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -351,6 +351,26 @@ about whether the bytes can be turned back into a working system, and a system c
|
||||
completely while having no executable recovery route for a given failure. That is not a hypothetical
|
||||
here — §8 has rows where it is the actual state.
|
||||
|
||||
### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
|
||||
|
||||
> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
|
||||
> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
|
||||
> byte-identical. **It passed with an operator present, and four manual interventions stood between
|
||||
> "the key is recoverable" and "the file is back" — none of which was in any design document.** They
|
||||
> are recorded here because this is the section a future reader will use to answer *"can the customer
|
||||
> do this alone?"*, and until 2026-08-05 the answer was no for reasons nothing wrote down.
|
||||
|
||||
| # | The step | Why it stopped a customer | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. |
|
||||
| 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed` → `OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. |
|
||||
| 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. |
|
||||
| 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** is required. | **OPEN → R-193.** This is a design decision (the one-shot credential model), not a defect, and it needs an operator ruling. **It is the reason the capability-map row keeps its "with a person present" qualifier.** |
|
||||
|
||||
> **[FACT] The honest current answer:** three of the four are gone. **A customer alone still cannot
|
||||
> complete a rebuild recovery**, because step 4 requires the operator. Everything after step 4 —
|
||||
> claim, restore, verify — is now self-service.
|
||||
|
||||
### 7.1 The dependency graph
|
||||
|
||||
> **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -59,14 +59,14 @@
|
||||
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
|
||||
| R-198 | **The superseded-escrow retention kept the K-escrow and dropped the identity blob** — so it preserved the PBS key and destroyed the offsite restic repository password, with the escrow CEREMONY as the destroying act | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | Corrects the escrow-ceremony capability-map row's v0.60.0 retention claim (was struck through 2026-08-04, now restated as what ships). One column + one SELECT list, in the shared `demoteCurrentEscrowTx`, so both callers are fixed at once. Nothing backfillable. **Load-bearing, not tidy:** with R-193(c) refused, customer-present recovery is the only route back from a rebuild and it runs entirely through the retained identity blob |
|
||||
| R-197 | **The hub stored both halves of "did this box's offsite data key change" and compared them nowhere** | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | New `offsite_repo_key_changed` (warning, operator-only, edge-triggered, no hash values). Flips nothing in the capability map — it adds a signal, not a capability |
|
||||
| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **comments corrected (hub v0.93.0); the BEHAVIOUR stays open** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping |
|
||||
| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **CLOSED — behaviour shipped hub v0.95.0 (2026-08-05)** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping **Collapsed 2026-08-05.** The comment correction landed in v0.93.0; the BEHAVIOUR landed in v0.95.0 with R-204 item 2 — the precautionary `MarkEscrowStale` and its `escrow_stale` event are gone from `ReissueCredentials`. Removing a signal was safe because the real case is measured twice over, and the mark was **blinding** one of them: a stale flag empties `restic_pw_sha256` from the ACK, which is what the controller's per-ACK re-check compares against. |
|
||||
| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
|
||||
| R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 6–8 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
|
||||
| R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
|
||||
| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
|
||||
| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
|
||||
| R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
|
||||
| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **OPEN — the wall the 2026-08-04 night drill hit** | Blocks R-201's verdict. Fix is most likely R-196's (mark stale on evidence, not on a Re-issue) or letting a verified recovered key confirm the escrow without a ceremony |
|
||||
| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **ITEMS 1–3 SHIPPED (controller v0.198.0 + hub v0.95.0, 2026-08-05); ITEM 4 OPEN → R-193** | Collapsed per the coupling rule. The fix taken was R-196's — remove the guess, keep the two MEASURED signals (the controller's per-ACK hash re-check, which the stale flag was blinding, and R-197's supersession detector). Plus the claim-code read-through and the restore-scope wording. **Flips the capability-map recovery row from four crutches to one**, and that one is R-193's operator Re-issue, which keeps the row's *with a person present* qualifier. Does NOT touch R-198's retention, which is still unit-proven and needs the second drill |
|
||||
| R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet |
|
||||
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
|
||||
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
||||
|
||||
Reference in New Issue
Block a user