docs: R-196 closed, R-204 items 1-3 closed, item 4 open (R-193)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
- OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and its dependency stated. Header restates that R-202, the 1.2 GB ciphertext deletion and R-198's still-unit-proven retention all REMAIN OPEN. - capability map: the recovery row keeps its 'with a person present' qualifier, names which crutch remains, and cites the three now gone. - 07-backup-architecture: new 7.0 - what a customer can and cannot do ALONE, the four steps in a table with status. This is the section a future reader will use to answer that question. - CONTEXT: standing ruling S-32, superseding S-31 steps 2-5. - STATUS: rewritten to one screen per its own header; removes a corrupted half-overwritten section left from the drill session. - ROADMAP: R-196 and R-204 collapsed.
This commit is contained in:
1 parent
975a690fbe
commit
0dbd954fec
7 files changed
+272
-264
No files matched your search
+39
@@ -17,6 +17,45 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-32 — THREE OF S-31's FOUR MANUAL STEPS ARE CLOSED (2026-08-05, R-204 items 1–3 / R-196).
|
||||
controller v0.198.0 + hub v0.95.0. Read this BEFORE S-31 — it supersedes S-31's steps 2–5.**
|
||||
|
||||
- **Step 1 (the reset code) — CLOSED.** `web.effectiveClaimCode` now calls `settings.ReloadClaimCode()`
|
||||
before applying the settings-vs-config precedence, so a code minted by the SEPARATE
|
||||
`--print-reset-code` process is visible without a restart. **The precedence rule is unchanged** —
|
||||
the defect was the freshness of the settings value, not which source wins. It is a READ-THROUGH on
|
||||
the claim path only, deliberately **not** a watcher, a signal handler or a TTL: a TTL would leave a
|
||||
window in which a superseded code still works, which is worse than the bug. `effectiveClaimCode`
|
||||
now returns an error and **every caller fails closed** (`claimGateActive` keeps the gate UP; an
|
||||
absent settings file is NOT an error — a pre-first-save box legitimately falls back to the
|
||||
controller.yaml bake). **S-31's "restart the controller between minting and claiming" is obsolete.**
|
||||
- **Steps 2–4 (the false staleness) — CLOSED.** `offsite.ReissueCredentials` no longer calls
|
||||
`MarkEscrowStale` and no longer emits `escrow_stale`; `offsite_reissued` is untouched. **The
|
||||
mechanism that made this expensive, recorded because it is not obvious:** `stale_at` makes
|
||||
`GetEscrowStatusForCustomer` WITHHOLD `restic_pw_sha256` — which is the exact value the controller's
|
||||
Scenario-F re-check (`report.EscrowAutoConfirmer.reconcileEscrowed`) compares against its live repo
|
||||
password on every ACK. **The precautionary mark was blinding the measurement that would have told
|
||||
the truth**, leaving the box able to report only the hash-LESS reason, which is false. Coverage
|
||||
after removal: that per-ACK comparison (continuous, box-side) plus R-197's `offsite_repo_key_changed`
|
||||
(edge-triggered, hub-side, at a supersession). **`MarkEscrowStale` is KEPT with no caller** — the
|
||||
flag is still read by the ACK, the operator card and the PBS-DR view, and the right way to set it is
|
||||
a future caller that has MEASURED a change; `TestEscrowStaleMechanism_StillWithholdsAndClears` stops
|
||||
it decaying meanwhile. **The manual `/backup/offbox/confirm-escrow` workaround is no longer needed
|
||||
after a Re-issue.**
|
||||
- **Step 5 (the restore default) — CLOSED.** `restoreScratchOutcomeMsg` states, for `mode=unit`, what
|
||||
came back, that the customer's own files did NOT, and the step that gets them; the wizard's intent
|
||||
card 1 states its scope before the choice. **The `mode=full` size gate is untouched** and pinned as
|
||||
unchanged. **The default stays `unit`:** all three wizard forms set `mode` explicitly, so changing
|
||||
it would alter nothing the customer sees while silently changing a hand-crafted mode-less POST.
|
||||
- **WHAT REMAINS is S-31's step 1-before-the-list: a rebuilt box still cannot obtain an off-site
|
||||
credential unaided** (the one-time password was consumed by its predecessor), so an operator
|
||||
Re-issue is still required. That is **R-193**, it needs an operator ruling, and it is why the
|
||||
capability-map recovery row keeps its *"with a person present"* qualifier.
|
||||
- **Live-validated on demo boxes, endpoint-level** (no browser on DooPlex): reset codes on
|
||||
demo-felhom 9201 with `restarts=0` across the mints; the unit-restore outcome and the full-restore
|
||||
size-gate reveal on demo-hp 9201, using `privatebin` so the drill's `calibre-web` scratch was not
|
||||
touched. **The Part 2 change was NOT fired live on demo-hp** — a Re-issue there was out of scope.
|
||||
|
||||
**S-31 — THE DRILL PASSED: a customer's file survives a machine rebuild and comes back byte-identical.
|
||||
The capability is proven; the customer JOURNEY is four undocumented manual steps (2026-08-04, R-201/R-204).**
|
||||
|
||||
|
||||
@@ -1,124 +1,160 @@
|
||||
# REPORT — R-201: **THE DRILL PASSED.** A customer's file survived a machine rebuild
|
||||
# REPORT — R-196 / R-204 item 2 (hub v0.95.0), 2026-08-05
|
||||
|
||||
**2026-08-04, 21:30–23:15** · `demo-hp` deliberately rebuilt · no code, no version bump ·
|
||||
`demo-felhom` untouched. Record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.
|
||||
**A re-issue no longer marks a healthy escrow stale.** One behaviour change, one register closed, and
|
||||
the coverage claim proved rather than assumed. The controller's half of R-204 (items 1 and 3) is
|
||||
`felhom-controller` v0.198.0.
|
||||
|
||||
---
|
||||
## 1. Baselines, re-read on arrival
|
||||
|
||||
## 1. THE VERDICT — PASS
|
||||
| Repo | Expected | Found |
|
||||
|---|---|---|
|
||||
| `felhom.eu` | `2a7ac03c4726` / hub v0.94.0 (deployed `felhom-hub:0.94.0`) | **exact match**, tree clean, `HEAD == origin/main` |
|
||||
|
||||
```
|
||||
pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||
restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||
```
|
||||
**§3.2's landmark had DRIFTED, and the drift changed the work.** The task described
|
||||
`offsite.go:222-231` under a known-consequence comment saying the mark was made on a false premise.
|
||||
That comment had already been rewritten by the R-196 comment-correction commit, and the version on
|
||||
`main` gave a **non-false** ground for the mark: *"the box's re-apply may mint a fresh repository
|
||||
password (it does exactly that whenever `<DataDir>/offbox/repo_password` is absent — the
|
||||
guest-rebuild shape)"*. So the question was no longer "delete a comment's lie" but "is the shape it
|
||||
guards actually covered elsewhere?" — which is Scenario D, and §8.2 says to stop and report if it is
|
||||
not. It is; §3 below is the evidence.
|
||||
|
||||
**Byte-identical.** The controller's data volume was destroyed and the file deleted from disk; the
|
||||
customer's recovery code, the customer's restore flow, and the file came back. **The Felhom backup
|
||||
story is proved end to end for the first time.**
|
||||
## 2. What changed
|
||||
|
||||
## 2. Snapshot count at step 9 — **3, not 1**
|
||||
`offsite.ReissueCredentials` no longer calls `MarkEscrowStale` and no longer emits the `escrow_stale`
|
||||
event. **`offsite_reissued` is untouched** and still fires on every re-issue. The known-consequence
|
||||
comment is rewritten to record what was done, when, and why — with the disagreement below stated in
|
||||
it rather than absorbed.
|
||||
|
||||
`repo_state: null` (the repository **opened**, not orphaned), `snapshot_count: 3`, `repo_size_bytes:
|
||||
42 026` — the pre-wipe size exactly. The pre-wipe snapshot `e6132ae5` was still present with the file
|
||||
inside it. **No fresh history was started over the old one** — the failure this drill was designed to
|
||||
catch did not occur.
|
||||
**What the mark actually cost, established mechanically rather than asserted** (this is why it was a
|
||||
blocker and not a nit):
|
||||
|
||||
## 3. §5's five conditions, recorded before the wipe
|
||||
1. `stale_at` set → `GetEscrowStatusForCustomer` **withholds** `restic_pw_sha256` from the report ACK.
|
||||
2. With no hash, the controller's SLICE-3 auto-confirm returns early and cannot flip
|
||||
`pending → escrowed`.
|
||||
3. `OffboxRunnable() = OffboxConfigured() && EscrowState == "escrowed"` → **every off-site backup
|
||||
refused**, indefinitely, on a box whose repository key was never in doubt.
|
||||
4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
|
||||
supersedes the sealed blob. **During a recovery that is the one act that would have destroyed the
|
||||
key just recovered.**
|
||||
|
||||
1. sentinel **listed by name** in snapshot `e6132ae5` (19:36:26), 181 B.
|
||||
2. rollback archive `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` (1 606 765 083 B) **verified by a
|
||||
full zstd stream read** (4 867 573 760 B) with the sentinel confirmed inside.
|
||||
3. §3's option — §4 below.
|
||||
4. `demo-felhom` `health=ok`, `escrow_state=escrowed`, untouched throughout.
|
||||
5. nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.
|
||||
## 3. Scenario D — the evidence that the removed marking is covered
|
||||
|
||||
**One precondition had drifted and was repaired, not worked around:** the staged snapshot had lost the
|
||||
sentinel because the afternoon's experiment produced a *later* same-day snapshot and `forget
|
||||
--keep-daily 7 --group-by host,tags` pruned the good one. One backup re-established it, re-verified by
|
||||
listing. **A good snapshot is not durable against a later bad run on the same day.**
|
||||
The mark was precautionary and aimed at ONE shape: a re-issue followed by a box that mints a fresh
|
||||
repository password (the guest-rebuild shape, where `offbox/repo_password` is absent). That shape is
|
||||
measured in two independent places, and **the mark was blinding one of them**:
|
||||
|
||||
## 4. §3 — the recovery code
|
||||
- **Continuous, box-side — the real coverage.** `report.EscrowAutoConfirmer.reconcileEscrowed`
|
||||
(controller) compares the ACK's sealed `restic_pw_sha256` against the box's CURRENT local repo
|
||||
password on **every report ACK**, raising the stale flag, the customer card and the
|
||||
„create a new recovery code" CTA on a mismatch. That is a **measurement**, not a guess, and it is
|
||||
continuous rather than edge-triggered. Pinned by the controller's
|
||||
`TestEscrowStale_MismatchWarnsOnceAndFlags` — re-run green this session.
|
||||
**And step 1 above was blinding it:** a stale flag empties the very hash that comparison needs, so
|
||||
the box could only reach the hash-LESS branch and report *"the hub's current blob carries NO
|
||||
password hash"* — which is false. Removing the mark restores the true signal.
|
||||
- **Edge-triggered, hub-side.** R-197's `offsite_repo_key_changed` fires from
|
||||
`api.maybeEmitRepoKeyChanged` on a proven hash difference across a supersession and pages the
|
||||
operator. **Red-proved:** removing the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut`
|
||||
made `TestEscrowPut_ChangedRepoKey_RaisesSignal` fail with *"the repository key demonstrably changed
|
||||
and NO signal was raised"*, while the two silence tests stayed green.
|
||||
|
||||
**Option B as already in place, improved: no new copy was made, so nothing needed shredding.** The
|
||||
operator's own `~/.config/credentials` (`0600`) was the source; the code was **piped to stdin** for
|
||||
each invocation — never an argument, never exported, never a second file, never logged. Destroying the
|
||||
operator's own store would have destroyed their record; because no extra copy existed, there is nothing
|
||||
left to prove gone.
|
||||
**Disagreement recorded, per the R-96 standing rule.** Scenario D as written asks that a real key
|
||||
change also *"mark the escrow stale"*. **It must not, and nothing was changed to make it.** The hub
|
||||
learns of a real change at the instant a supersession **seals the new password** — i.e. when the
|
||||
escrow is at its freshest. Marking it stale there would ask the customer for a ceremony to fix the
|
||||
ceremony that just ran. The correct consequence at that instant is the operator alarm, which is
|
||||
exactly what R-197 already does. This is recorded in the code comment, the CHANGELOG and OPEN-ITEMS,
|
||||
not only here.
|
||||
|
||||
**Verified anyway, with the planted-copy positive control:** 0 hits in the agent journal, 0 in the
|
||||
controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover
|
||||
`felhom-idesc-*` dirs — and the same sweep found a planted copy (**1**), then **0** after shredding it.
|
||||
## 4. `MarkEscrowStale` is kept with no caller — deliberately
|
||||
|
||||
## 5. Every step's observable
|
||||
Per task §5 it was not to be modified, and it is not deleted either. The `stale_at` flag remains live
|
||||
and correct — read by the ACK, the operator config card and the PBS-DR view — and the right way to
|
||||
set it is a **future EVIDENTIAL caller** that has measured a key change rather than guessed at one.
|
||||
Its doc comment now says so plainly instead of naming a caller that no longer exists, and
|
||||
`TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying to inert while
|
||||
nothing writes it (the seam-built-but-never-wired shape, in reverse).
|
||||
|
||||
| step | observable |
|
||||
The schema comment and `EscrowStatus.Stale`'s comment were corrected the same way — each of the three
|
||||
previously asserted a writer that is now gone.
|
||||
|
||||
## 5. Files modified
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| 6 | fresh data dir `20:00:2x`; new `encryption.key`; `claimed = None`; `offbox = null` |
|
||||
| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED.** Nothing re-escrowed itself |
|
||||
| **8** | **recovered `8a9e33aa4da6…`** — matches the pre-wipe on-disk key and the hub's record |
|
||||
| 9a | `[INSTALLED] … reads back identical` |
|
||||
| 9b | after Re-issue + apply, the on-disk key is **still** the recovered one |
|
||||
| 9c | **repository OPENED** — `repo_state: null`, 3 snapshots, 42 026 B |
|
||||
| **10** | **restored sha256 byte-identical** |
|
||||
| 11 | this record |
|
||||
| `hub/internal/offsite/offsite.go` | the pessimistic `MarkEscrowStale` + `escrow_stale` event removed; comment rewritten to record the change, the coverage and the disagreement |
|
||||
| `hub/internal/offsite/offsite_test.go` | `TestReissue_InvalidatesEscrow` **replaced in place by its exact inverse** `TestReissue_DoesNotMarkAHealthyEscrowStale`; new `TestEscrowStaleMechanism_StillWithholdsAndClears` |
|
||||
| `hub/internal/store/store.go` | three comments corrected (`MarkEscrowStale`, the `stale_at` schema note, `EscrowStatus.Stale`) — each had named a writer that no longer exists |
|
||||
| `manifests/hub.yaml` | image tag `0.94.0` → `0.95.0` |
|
||||
| `hub/CHANGELOG.md`, `CONTEXT.md`, `STATUS.md`, `documentation/…` | v0.95.0 entry; ruling **S-32**; the register and architecture updates below |
|
||||
|
||||
Superseded escrow rows remained **2** throughout — **no ceremony was run at any point.**
|
||||
**Commits on `main`:** `d1a8edb` (behaviour + tests + comments) · `5c7d671` (CHANGELOG) ·
|
||||
`975a690` (manifest bump).
|
||||
|
||||
## 6. Part 2 — not run
|
||||
**Deploy:** built + pushed `felhom-hub:0.95.0`, bumped `manifests/hub.yaml`, pushed, then a
|
||||
**deliberate ArgoCD hard-refresh + sync** (auto-sync stays off; no `kubectl set image` anywhere).
|
||||
Result: app `felhom` **Synced / Healthy**, `deploy/hub` rolled out, running image
|
||||
`gitea.dooplex.hu/admin/felhom-hub:0.95.0`, startup log clean (offsite provisioning, pool-box checker,
|
||||
PBS-DR reconciler and all six host checkers initialised; `Listening on :8080`).
|
||||
|
||||
Gate: *"the drill PASSED; its evidence is written down; and there is time."* The first two are met; the
|
||||
third is not — 23:15, after a full destructive cycle, and Part 2 is a second wipe. Deliberately left
|
||||
for a fresh session with drill 1's result already recorded, so a second wipe cannot overwrite it.
|
||||
**R-198's retention remains unit-proven only** — nothing has yet superseded a key in production, and it
|
||||
is now the last unproven link in this chain.
|
||||
## 6. Tests and red-proofs
|
||||
|
||||
## 7. Teardown — three layers
|
||||
Green gate: `cd hub && go build ./... && go vet ./... && go test ./...` — **rc=0**.
|
||||
`python3 scripts/repo_gates.py --fast` — **all five gates OK**.
|
||||
|
||||
| layer | state |
|
||||
|---|---|
|
||||
| the guest | **healthy and re-armed**: controller v0.197.0 claimed, six app containers serving, all three apps re-toggled for off-site, the sentinel restored to its live location and verified. The verification-copy tree is **kept as evidence**. Scratch band empty |
|
||||
| the host | vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted |
|
||||
| the hub | **no new customer records** — `demo-hp` is the same row throughout (a controller-data rebuild, not a re-enrolment). One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. **The two superseded escrow rows are unchanged** |
|
||||
| Test | Result | Red-proof — what was mutated | Outcome |
|
||||
|---|---|---|---|
|
||||
| `TestReissue_DoesNotMarkAHealthyEscrowStale` (C) | PASS | restored the pessimistic `MarkEscrowStale` block in `ReissueCredentials`, exactly as it was | **FAILED** — *"a re-issue marked a HEALTHY escrow stale…"* |
|
||||
| `TestEscrowStaleMechanism_StillWithholdsAndClears` | PASS | same mutation | **stayed GREEN** — correctly: the mutation restores a *caller*, not a break in the mechanism. That split is the evidence Scenario C's assertion is about the caller and not the flag. |
|
||||
| `TestEscrowPut_ChangedRepoKey_RaisesSignal` (D) | PASS | removed the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut` | **FAILED** — *"the repository key demonstrably changed and NO signal was raised"* |
|
||||
| `TestEscrowPut_UnchangedRepoKey_Silent`, `TestEscrowPut_HashlessSupersession_NoSignal` | PASS | same | stayed green — the detector's silence branches are independent |
|
||||
| controller `TestEscrowStale_MismatchWarnsOnceAndFlags` | PASS | — (cited as the continuous-coverage pin) | — |
|
||||
|
||||
**Nothing deleted on the storage endpoint**, including the ~1.2 GB of orphaned ciphertext — ruled,
|
||||
still owed, deliberately not ridden along with a drill.
|
||||
Scenario C asserts the **consequence** (the ACK still carries the hash, so auto-confirm can proceed)
|
||||
rather than the mechanism (that a function was not called), because the hash is what the drill's
|
||||
blockage actually turned on. It also asserts that `offsite_reissued` still fires — removing a false
|
||||
alarm must not remove the true notice.
|
||||
|
||||
## 8. The capability-map row
|
||||
## 7. Live validation
|
||||
|
||||
Now reads *"A customer's file survives a machine rebuild and comes back — the whole off-site story, end
|
||||
to end"*, **PROVEN-LIVE**. What it deliberately does **not** claim: the journey is manual and
|
||||
undocumented (R-204); the scope is `demo-hp` and a **controller-data** rebuild, **not** a total host
|
||||
loss and **not** a guest reprovision; and R-198's retention is still unit-proven.
|
||||
**Per task §12 point 5, a live re-issue was NOT run, and must not have been on demo-hp** — it would
|
||||
have been a credential rotation on the box holding the drill's evidence. Part 2 is proved by test and
|
||||
by the deployment being live and healthy. The controller-side halves of R-204 were validated live and
|
||||
are reported in `felhom-controller/REPORT.md`.
|
||||
|
||||
## 9. New findings — R-204, expanded into the gap list
|
||||
## 8. Register and documentation
|
||||
|
||||
The four steps between a recovered key and a restored file, all measured while walking them:
|
||||
- **`OPEN-ITEMS.md`** — **R-196 → CLOSED (hub v0.95.0)**; **R-204 → items 1–3 CLOSED, item 4 OPEN
|
||||
(→ R-193)** with its dependency named. The header block is updated and states explicitly that
|
||||
**R-202**, **the ~1.2 GB orphaned-ciphertext deletion** and **R-198's retention (still UNIT-PROVEN
|
||||
ONLY — the second deliberate wipe is the next item)** all **remain open**, so nothing is presumed
|
||||
closed by association. R-201 is recorded as PASSED. **R-204 is still the highest ID; nothing new
|
||||
was minted.**
|
||||
- **`architecture/00-capability-map.md`** — the recovery row now says three of the four crutches are
|
||||
gone, names the fixes and their evidence, and states that **item 4 (R-193) is the one that remains**
|
||||
and is why the row **keeps its "with a person present" qualifier**. The **R-199 back-pointer was
|
||||
already present** on the adjacent key-recovery row (added when that row was last corrected), so it
|
||||
needed no further action — verified, not assumed.
|
||||
- **`architecture/07-backup-architecture.md`** — **new §7.0, "What a customer can and cannot do
|
||||
ALONE"**: the four steps in a table with what each cost and its status, plus the honest current
|
||||
answer. This is the section a future reader will use to answer the question.
|
||||
- **`documentation/backlog/ROADMAP.md`** — R-196 and R-204 collapsed per the coupling rule.
|
||||
- **`CONTEXT.md`** — new standing ruling **S-32**, which supersedes S-31's steps 2–5 and carries the
|
||||
blinding mechanism, the fail-closed rule and the "no TTL" reasoning forward.
|
||||
- **`STATUS.md`** — rewritten to **one screen** (191 → ~90 lines) per its own header. It also had a
|
||||
corrupted, half-overwritten "What we're working on" section left from the drill session, which is
|
||||
now gone. Next item stated as the retention drill.
|
||||
|
||||
1. **R-193** — a rebuilt controller cannot configure its off-site tier (`no unconsumed offsite
|
||||
password`; the ledger shows its predecessor consumed it at 07:12:06). Needs an operator Re-issue.
|
||||
2. **The claim gate** — a rebuilt box is unclaimed, so *every* controller endpoint is intercepted. And
|
||||
**the local escape hatch does not work unaided**: `--print-reset-code` writes the new hash to
|
||||
`settings.json` while the running controller keeps its old copy in memory, so `effectiveClaimCode()`
|
||||
never sees it. **Restart the controller between minting and claiming** — two claim attempts failed
|
||||
before this was diagnosed.
|
||||
3. **R-196** — the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged, which
|
||||
gates every off-site run. Cleared with the **manual** confirm, never a ceremony.
|
||||
4. **`mode=unit` is the restore default** and returns the recovery unit, **not** the userdata leg. A
|
||||
customer told to "restore from off-site" gets their app definition and not their documents, and
|
||||
nothing in the outcome says so. **The worst of the four**, because it fails silently at the last
|
||||
step.
|
||||
**CI:** felhom.eu runs **154** (`5c7d671`, code) and **155** (`975a690`, manifest) — both success.
|
||||
`--no-verify` was **not** used; the pre-push gate ran and passed on every push.
|
||||
|
||||
## 10. CI
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
Docs push only; run number and task id in the session summary. **`--no-verify` not used.**
|
||||
|
||||
## 11. Observations — noticed, NOT acted on
|
||||
|
||||
1. **A same-day re-run replaces the day's snapshot** (`forget --keep-daily 7 --group-by host,tags`).
|
||||
Any drill depending on a specific snapshot surviving must account for it.
|
||||
2. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
|
||||
`kubectl port-forward`; the memory's `curl -u :$HUB_PW` recipe omits that.
|
||||
3. **`--print-reset-code` mints a new generation on every invocation** — three wasted generations here
|
||||
(4, 5, 6) while diagnosing the in-memory staleness.
|
||||
4. **The restore wrote into `backups/offsite-restore/<app>/` mirroring the full absolute path** — deep
|
||||
but unambiguous; worth knowing before writing customer-facing copy about where files land.
|
||||
- **`allowedEventTypes` still lists `escrow_stale`**, which after this change has **no producer** in
|
||||
either repo. It is inert rather than harmful; removing an allowlist entry is a behaviour change and
|
||||
is out of this session's scope.
|
||||
- `MarkEscrowStale` is now dead code by call-graph. Kept on purpose (§4 above) — but if a future
|
||||
session's linter or cleanup pass proposes deleting it, the reason it exists is in its doc comment
|
||||
and in the test that exercises it.
|
||||
- `/` on DooPlex is at **86%** used — under the 90% abort line, but worth watching before large builds.
|
||||
@@ -1,6 +1,6 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Updated 2026-08-04.**
|
||||
**Updated 2026-08-05.**
|
||||
|
||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
||||
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
||||
@@ -13,179 +13,78 @@
|
||||
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
||||
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
||||
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
||||
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
||||
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
|
||||
box six times, everything returned every time, and an app switched off deliberately stayed off.
|
||||
Proven end to end on real hardware.
|
||||
places — the machine's drive, a second drive, and an encrypted off-site copy.
|
||||
|
||||
**And the whole backup promise is now proved.** On 4 August we destroyed a machine on purpose and
|
||||
deleted a marked file from its disk. Using the recovery code you saved: the backup key came back
|
||||
**identical, character for character**; the existing off-site store **opened** rather than starting
|
||||
over; and the file was restored **byte for byte identical**. *(R-201)*
|
||||
|
||||
## What's broken
|
||||
|
||||
- **A customer still cannot do that recovery alone — but only one step is left.** Getting from "the
|
||||
key is recoverable" to "the file is back" took four steps that appeared in no instructions. **Three
|
||||
are fixed today** *(R-204)*: the local reset-code tool works on the first try instead of needing the
|
||||
controller restarted; re-issuing the storage credential no longer falsely marks the recovery key
|
||||
"stale" (which used to stop every off-site backup and invite the one act that would have destroyed
|
||||
the recovered key); and the everyday restore now **says in plain Hungarian that it returned the
|
||||
app's settings and database and not your documents**, and names the button that does. **The step
|
||||
that remains is the first one:** a rebuilt machine cannot get a storage credential by itself,
|
||||
because the one-time password was used up by its predecessor — so you still have to press
|
||||
Re-issue. That is a design decision waiting on you, below. *(R-193)*
|
||||
- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
|
||||
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one, so the old
|
||||
backups can no longer be opened by the machine that wrote them. Both demo machines did this on 3–4
|
||||
August — demo-hp lost access to 15 backups (41 MB), demo-felhom to 36 (1.14 GB), and **the second one
|
||||
happened in silence for thirteen hours**. From today that silence is fixed (below) and the old key
|
||||
survives the ceremony (below), but **the rebuild itself still starts a fresh history**, and getting
|
||||
the old one back is a path nobody has walked. *(R-193)*
|
||||
- *(fixed 4 Aug)* ~~The safety net we believed was under all of this is not there.~~ **The central hub
|
||||
was keeping the wrong key, and the fix shipped the same day.** When a machine re-seals its backups
|
||||
under a new recovery code, the hub keeps the old sealed package — but it was keeping the one for the
|
||||
whole-machine backups and **not** the one for the off-site file backups, which is the whole point. And
|
||||
because the swap happens in one motion, **the act that destroyed the old key was the recovery-code
|
||||
ceremony itself** — the exact thing a rebuilt machine asks its customer to do, on a screen promising
|
||||
the old backups stay recoverable. Both keys are now kept. **What this cannot undo:** anything
|
||||
superseded before today is gone for good, which includes both demo machines' pre-4-August keys — so
|
||||
those 51 orphaned backups were beyond reach even if the recovery codes had been kept. *(R-198)*
|
||||
- **IT WORKS. A customer's file survived a machine being destroyed and came back — proved last night on
|
||||
real hardware, end to end.** We wiped the HP machine's controller data on purpose and deleted the
|
||||
marked file from its disk. Using the recovery code you saved: the backup key came back **identical,
|
||||
character for character**; the existing off-site store **opened** rather than starting over (the same
|
||||
three backups, the same 42 026 bytes as before the wipe); and the file was restored **byte for byte
|
||||
identical**. That is the whole backup promise, demonstrated for the first time. *(R-201)*
|
||||
- **But a customer could not have done it alone, and that is now the open work.** Getting from "the key
|
||||
is recoverable" to "the file is back" took four steps that appear in no instructions: re-issuing the
|
||||
storage credential; re-claiming the machine (whose local reset-code tool **does not work until the
|
||||
controller is restarted** — two attempts failed before we spotted it); manually clearing a "stale"
|
||||
mark that the re-issue sets even though the key never changed; and choosing the *full* restore,
|
||||
because the default one returns the app's settings and **not** the customer's documents — with
|
||||
nothing saying so. **The capability is real; the experience is not built yet.** *(R-204)*
|
||||
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one. Both demo
|
||||
machines did this on 3–4 August — 51 backups (~1.2 GB) between them. The old key now survives the
|
||||
recovery ceremony, and a changed key now raises an alarm the same day, but **the rebuild itself
|
||||
still starts a fresh history**. *(R-193)*
|
||||
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
|
||||
says the old backups may later be restorable with the matching recovery code. From today that is true
|
||||
for machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||||
cannot tell which case it is in without the recovery path above. We deliberately did **not** patch the
|
||||
sentence: a conditional promise that can still be wrong is worse on that screen than a vague one.
|
||||
*(R-202)*
|
||||
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
|
||||
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
|
||||
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
|
||||
it every day since 27 July. Both machines and all future ones. *(R-191)*
|
||||
says the old backups may later be restorable with the matching recovery code. That is true for
|
||||
machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||||
cannot tell which case it is in. We deliberately did **not** patch the sentence: a conditional
|
||||
promise that can still be wrong is worse there than a vague one. *(R-202)*
|
||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap.
|
||||
*(R-95, R-87)*
|
||||
also delete it. A daily snapshot is armed as a stopgap. *(R-95, R-87)*
|
||||
|
||||
## What shipped recently
|
||||
|
||||
- **The daily error email about David is gone.** David's record was created when he said he would try
|
||||
Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every
|
||||
morning at five the system reported that a database backup had not happened. **A customer with no
|
||||
machine ever attached now produces no alarm.** A machine that *was* attached and went quiet still
|
||||
does, because that is a real fault, and there is a test whose whole job is to make sure a future
|
||||
change cannot take that away. David's record was not touched — the record was right; the alarm was
|
||||
wrong. **The first quiet morning will be 5 August, so this is not confirmed yet.** *(R-195)*
|
||||
|
||||
- **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we
|
||||
found a permission that vanished with no reinstall and no recorded action; the machine now notices,
|
||||
restores it through the same controlled helper that grants it in the first place, and **raises an
|
||||
alert saying it was missing and was restored**. The alert is the point: a permission that silently
|
||||
fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo
|
||||
machine by taking the permission away by hand. *(R-190 — the repair; why it vanished is still open)*
|
||||
|
||||
- **The on-machine backup copy has now been proved to restore — by the machines themselves**, both of
|
||||
them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the
|
||||
off-site copy. *(R-86, R-185)*
|
||||
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
||||
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
||||
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
||||
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
|
||||
written to disk with the result and survives. This was caught happening, not predicted: a real
|
||||
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
|
||||
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
|
||||
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
||||
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
||||
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
||||
|
||||
- **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time;
|
||||
a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
|
||||
restore no longer quietly returns the wrong thing. *(R-204 items 1–3, R-196)*
|
||||
- **2026-08-04 (night)** — **The drill PASSED**, end to end, on real hardware. *(R-201)*
|
||||
- **2026-08-04** — The folder-left-out-of-the-backup problem fixed both halves: the app and its backup
|
||||
look in the same directory, and a backup that misses a folder marked essential reports *incomplete*
|
||||
instead of success. *(R-203)*
|
||||
- **2026-08-04** — The hub now keeps the off-site backup key when a machine re-seals, instead of only
|
||||
the whole-machine one, and a machine can fetch its own sealed package back. *(R-198, R-199)*
|
||||
- **2026-08-04** — The daily false alarm about David is gone; a vanished permission now repairs itself
|
||||
**and says it had to**; the weekly off-site backup stopped reporting failure after a successful
|
||||
upload. *(R-195, R-190, R-191)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** making that journey something a customer can actually follow — the four steps above, worst
|
||||
first: the restore default that silently returns the wrong thing. The HP machine is sitting
|
||||
mid-drill with the recovered key already on it; it needs re-claiming and one setting confirmed, then
|
||||
the backup runs and the file is restored. **Do not let it make a new recovery code** — that would
|
||||
The marked file now lands in
|
||||
the off-site backup, so there is finally something to recover. Everything else is already in place on
|
||||
the HP machine — the recovery code you saved, a working off-site store, the app and the file. Nothing
|
||||
has been wiped; that step waits for your go-ahead at a marked stop point. Both honesty fixes shipped today: a changed backup key now
|
||||
raises an alarm on the day, and the email that stated the opposite of what it measured now describes
|
||||
what it actually saw.
|
||||
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
|
||||
refuse and ask, rather than start fresh — but that is a prediction from the code, not something
|
||||
anyone has watched happen.
|
||||
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
||||
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
||||
*(R-87)*
|
||||
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
||||
- **Next: the retention proof.** The hub keeping the old sealed key when a machine re-seals is the one
|
||||
remaining link that has **never run outside a test**. Proving it needs a second deliberate wipe on
|
||||
the spare demo machine, and it is its own procedure. *(R-198)*
|
||||
- **Then:** the orphaned-backup deletion you asked for (below), and the off-site copy the machine can
|
||||
still erase. *(R-193, R-95)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- *(decided 4 Aug — recorded here until the work it unblocks is done)* You chose **not** to keep a
|
||||
copy of the backup key on the Proxmox host. That is a coherent choice and it has a consequence worth
|
||||
restating: **the customer's own recovery code is now the only route back from a rebuild**, so the
|
||||
sealed package the hub keeps is the single copy. Today's fix is what makes that true rather than
|
||||
wishful — until this morning the ceremony destroyed it. *(R-193)*
|
||||
- **Your ruling about the recovery screen has been priced, and it can be built.** You said: a freshly
|
||||
installed machine that finds a sealed package waiting should say so loudly, offer the customer a box
|
||||
to type their recovery code into, and show what would come back before doing anything. **All of it is
|
||||
buildable, and one part is already free** — the hub is *already* telling every machine, on every
|
||||
check-in, that a sealed package exists and which key it covers, and the machine currently throws that
|
||||
message away. Showing a preview is also cheap: listing what is in an off-site store reads it without
|
||||
writing to it, so the customer can see how many backups, from when, and for which apps before
|
||||
committing. The real work is one new connection: the unsealing has to happen in the part that runs on
|
||||
the Proxmox host, because the customer-facing part cannot do it — the same crossing your recovery-code
|
||||
ceremony already makes in the other direction. **One thing for you to weigh, which we deliberately did
|
||||
not decide:** a screen that takes a recovery code and then shows what is in a backup store is reachable
|
||||
by anyone with the household's dashboard password, and the preview reveals backup dates and app names.
|
||||
*(R-193)*
|
||||
- **Should we run the proof?** Nothing about recovery has ever actually been done, so we designed the
|
||||
exercise: on the spare demo machine, make a recovery code and **keep it**, put a marked file into an
|
||||
app, back it up off-site, wipe the machine, reinstall, recover with the saved code, and check the
|
||||
marked file comes back byte for byte. About half a day, of which under an hour needs you. The failure
|
||||
we are watching for is precise: **if the recovered machine reports one backup instead of the ones we
|
||||
put there, it started a fresh history and the proof failed** — "the store opened" is not good enough.
|
||||
It costs that machine's contents if it goes wrong, which is acceptable there. **You said yes, after
|
||||
R-198 — and R-198 shipped this morning, so this is ready to run on your word.** *(R-201)*
|
||||
- **The one-shot credential decision — this is now the last thing between a customer and an unaided
|
||||
recovery.** A rebuilt machine has no storage credential of its own, so an operator must press
|
||||
Re-issue. Everything after that point is now self-service. Deciding how a rebuilt machine should get
|
||||
a credential is the remaining design question. *(R-193, R-204 item 4)*
|
||||
- **The recovery screen you described has been priced, and it can be built.** A freshly installed
|
||||
machine that finds a sealed package waiting should say so, offer a box for the recovery code, and
|
||||
show what would come back before doing anything. One thing to weigh, deliberately not decided: that
|
||||
screen is reachable by anyone with the household's dashboard password, and the preview reveals
|
||||
backup dates and app names. *(R-193)*
|
||||
- **The orphaned backups on the storage box — you said delete, and it is still owed.** About 1.2 GB
|
||||
across the two demo machines, in set-aside stores nobody can open and nothing prunes. Deliberately not
|
||||
done today: erasing data on the storage endpoint is not something that should ride along with a
|
||||
database change, so it wants its own session. *(R-193)*
|
||||
across the two demo machines, in set-aside stores nobody can open and nothing prunes. It wants its
|
||||
own session rather than riding along with other work. *(R-193)*
|
||||
- *(decided 4 Aug)* You chose **not** to keep a copy of the backup key on the Proxmox host, which
|
||||
makes the customer's own recovery code the only route back from a rebuild. *(R-193)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
machines to install, only which ones exist. Closing that needs either a password given to the build
|
||||
server or a check inside the hub itself. *(R-184)*
|
||||
- **Nothing else.**
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-04 (night)** — **The drill PASSED.** We destroyed a machine on purpose and the customer's
|
||||
file came back byte-for-byte identical, using the recovery code you saved. First time the backup
|
||||
story has been proved end to end. It needed four undocumented manual steps to get there, which are
|
||||
now the next piece of work. The machine is up, healthy and re-armed. *(R-201, R-204)*
|
||||
- **2026-08-04 (evening)** — **Fixed the folder-left-out-of-the-backup problem, both halves.** The app
|
||||
and its backup now look in the same directory, and a backup that misses a folder marked essential
|
||||
reports *incomplete* instead of success. Proved by listing the backup's own contents and finding the
|
||||
marked file. The wipe-and-restore proof is unblocked. *(R-203, R-201)*
|
||||
- **2026-08-04 (late)** — Set up the wipe-and-restore proof on the HP machine and **stopped before the
|
||||
wipe**: a folder marked as protected was missing from the off-site backup while the backup reported
|
||||
success. Three things were proved on the way, all firsts: a rebuilt machine's off-site backup now
|
||||
**refuses and says so** instead of quietly starting over; the "start a new store" repair works and
|
||||
keeps the old data aside; and the HP machine's pre-3-August history is gone for good — its key was
|
||||
destroyed four hours before the fix that would have kept it. *(R-203, R-201, R-193)*
|
||||
- **2026-08-04 (evening)** — **Proved the backup key comes back.** A demo machine fetched its own
|
||||
sealed package, opened it with the saved recovery code, and produced a key identical to the one it
|
||||
uses. A wrong code was refused and wrote nothing. The recovery code left no trace anywhere on the
|
||||
machine — searched, with a planted copy first to prove the search worked. *(R-199, R-200)*
|
||||
- **2026-08-04 (earlier)** — **Fixed the worst of it.** The hub now keeps the off-site backup key when a
|
||||
machine re-seals, instead of only the whole-machine one — four lines of database change that stop the
|
||||
recovery ceremony from destroying the thing it exists to protect. A changed backup key now raises an
|
||||
alarm the same day. The daily email that described the opposite of what it measured now states what it
|
||||
saw, and a refusal to auto-repair leaves a record instead of silence. *(R-198, R-197, R-192, R-196)*
|
||||
- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
|
||||
password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
|
||||
access to their off-site history, one of them silently. A decision is now waiting on you. The daily
|
||||
false alarm about David is fixed. *(R-193, R-195)*
|
||||
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
||||
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
||||
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
||||
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
||||
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
||||
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|
||||
- **One small question, not urgent.** The automatic version check cannot see which version you have
|
||||
told machines to install, only which ones exist. *(R-184)*
|
||||
@@ -40,7 +40,7 @@
|
||||
|
||||
| **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. **R-199 BACK-POINTER (omitted when this row was written): the recovery chain's link inventory and the per-link status live in `audits/RECON-offsite-dr-chain-2026-08-04.md` §3; links 1–8 are walked, 9–11 are not.** **Re-confirmed 2026-08-04 evening by an attempt to prove the DATA half:** the R-201 drill was prepared on demo-hp and **halted before the wipe** — the sentinel file was not in the off-site snapshot (R-203), so the wipe would have destroyed it and proven nothing. **No file has still ever been restored from an off-site backup after a wipe** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). The install half of the chain (controller v0.196.0 `--recover-offsite-install`, R-200) is likewise unit-proven only — it has never run against a live recovery. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent Line truncated
|
||||
|
||||
| **A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end** | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE (2026-08-04 night drill)** | demo-hp's controller data volume was destroyed and the sentinel deleted from disk. The customer's recovery code then produced `8a9e33aa4da6…` — **byte-identical** to the pre-wipe on-disk key and to the hub's independent record — and installed cleanly on the bare box. `identity_blob` was unchanged across the wipe (572 B, `updated_at` still 11:11:37). Evidence: `audits/DRILL-r201-night-run-2026-08-04.md` §2 | **WHAT IT DOES AND DOES NOT CLAIM.** PROVEN: after a real rebuild the key recovers byte-identical, the EXISTING repository opens (3 snapshots, 42 026 B — the pre-wipe size; not a fresh history), and the customer restore flow returns the file byte-identical. **NOT PROVEN, and stated because the drill needed all four:** the journey is **manual and undocumented** — an operator Re-issue (R-193), a re-claim whose escape hatch needs a controller restart to work at all, a manual escrow confirm to undo R-196's false staleness, and `mode=full` on the restore because the default returns the app definition and not the customer's files. **A customer doing this unaided today would not get their data back** (R-204). Scope: `demo-hp`, a controller-data-volume rebuild — **NOT** a total host loss, and **NOT** a guest reprovision. R-198's retention is still unit-proven: nothing has superseded a key in production |
|
||||
| **A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end** | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE (2026-08-04 night drill)** | demo-hp's controller data volume was destroyed and the sentinel deleted from disk. The customer's recovery code then produced `8a9e33aa4da6…` — **byte-identical** to the pre-wipe on-disk key and to the hub's independent record — and installed cleanly on the bare box. `identity_blob` was unchanged across the wipe (572 B, `updated_at` still 11:11:37). Evidence: `audits/DRILL-r201-night-run-2026-08-04.md` §2 | **WHAT IT DOES AND DOES NOT CLAIM.** PROVEN: after a real rebuild the key recovers byte-identical, the EXISTING repository opens (3 snapshots, 42 026 B — the pre-wipe size; not a fresh history), and the customer restore flow returns the file byte-identical. **STILL PROVEN ONLY WITH A PERSON PRESENT — and that qualifier stays on this row.** The drill needed FOUR manual interventions between "the key is recoverable" and "the file is back", none of them in any design document (R-204). **THREE ARE NOW CLOSED (2026-08-05, controller v0.198.0 + hub v0.95.0):** the re-claim escape hatch no longer needs a controller restart — a freshly minted code is accepted on the first attempt and a superseded one dies the instant the next is minted (R-204 item 1, proven live on demo-felhom 9201); a credential Re-issue no longer marks a healthy escrow stale, so it no longer blocks every off-site run nor invites the ceremony that would have destroyed the recovered key (R-196 / item 2 — test-proven, deliberately NOT fired on demo-hp); and the default `mode=unit` restore now STATES that it returned the app's definition and database and NOT the customer's files, naming the step that does (item 3, proven live on demo-hp: *„A saját fájljaid (dokumentumok, képek, feltöltések) NEM kerültek vissza…”*). **THE ONE THAT REMAINS is item 4: a rebuilt box cannot obtain an off-site credential unaided** — the one-time password was spent by its predecessor, so an operator Re-issue is still required. That is **R-193**, it needs an operator ruling, and until it is closed this row cannot lose its qualifier. Scope: `demo-hp`, a controller-data-volume rebuild — **NOT** a total host loss, and **NOT** a guest reprovision. R-198's retention is still unit-proven: nothing has superseded a key in production |
|
||||
| **Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts** | controller v0.197.0 | **PROVEN-LIVE (2026-08-04) — and this row was OPTIMISTIC before it** | On demo-hp, an app on the **system-data fallback** bound `/mnt/sys_drive/userdata/media/books` while the capture set looked in `/mnt/sys_drive/felhom-data/userdata/media/books`: the declared-mandatory directory was in **no** snapshot and the run reported `ok` (R-203). After v0.197.0 the capture log reads `1 mandatory path(s)` and the file is **listed inside the snapshot** — `restic ls -l latest --tag calibre-web` → `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt`. Evidence: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 and the v0.197.0 CHANGELOG | **WHAT IT DOES NOT CLAIM.** It covers CAPTURE, not RESTORE: **no file has ever been restored from an off-site backup after a wipe** (R-201, ready to resume). It also does not claim the enrolled-drive layout was ever wrong — it was not, which is precisely why this survived. And a run that still misses a mandatory directory now reports **`incomplete`** rather than `ok`, so this row's guarantee is one the status can express || **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** |
|
||||
| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **THE HOST TIER IS NOW PROVEN-LIVE, UNATTENDED, ON BOTH DEMO BOXES (2026-08-0Line truncated
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below |
|
||||
|
||||
@@ -351,6 +351,26 @@ about whether the bytes can be turned back into a working system, and a system c
|
||||
completely while having no executable recovery route for a given failure. That is not a hypothetical
|
||||
here — §8 has rows where it is the actual state.
|
||||
|
||||
### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
|
||||
|
||||
> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
|
||||
> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
|
||||
> byte-identical. **It passed with an operator present, and four manual interventions stood between
|
||||
> "the key is recoverable" and "the file is back" — none of which was in any design document.** They
|
||||
> are recorded here because this is the section a future reader will use to answer *"can the customer
|
||||
> do this alone?"*, and until 2026-08-05 the answer was no for reasons nothing wrote down.
|
||||
|
||||
| # | The step | Why it stopped a customer | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. |
|
||||
| 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed` → `OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. |
|
||||
| 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. |
|
||||
| 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** is required. | **OPEN → R-193.** This is a design decision (the one-shot credential model), not a defect, and it needs an operator ruling. **It is the reason the capability-map row keeps its "with a person present" qualifier.** |
|
||||
|
||||
> **[FACT] The honest current answer:** three of the four are gone. **A customer alone still cannot
|
||||
> complete a rebuild recovery**, because step 4 requires the operator. Everything after step 4 —
|
||||
> claim, restore, verify — is now self-service.
|
||||
|
||||
### 7.1 The dependency graph
|
||||
|
||||
> **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below
|
||||
|
||||
@@ -32,14 +32,28 @@ hub v0.94.0 + agent v0.125.0 + controller v0.195.0:**
|
||||
(`c60c8bc737a6…` from three independent sources: the box's file, the recovered bundle, and the hash
|
||||
the hub already stored).
|
||||
- **R-200 — the plumbing half shipped**, the customer-facing form did not, deliberately.
|
||||
- **R-201 — still OPEN, and the distinction is the point:** the KEY is proven recoverable; a
|
||||
recovered password has still never been INSTALLED, no repository has been reopened under one, and
|
||||
**no file has ever been restored**. The drill's pass condition is unchanged.
|
||||
- **R-201 — PASSED 2026-08-04 (night run).** A customer's file survived a machine rebuild and came
|
||||
back byte-identical, through the customer's own restore flow. **It passed only because a person was
|
||||
there:** four manual interventions stood between the recovered key and the restored file, none of
|
||||
them in any design document → R-204.
|
||||
- **R-202 — untouched.** The orphan card still promises recoverability unconditionally.
|
||||
- **The orphaned ciphertext deletion (~1.2 GB) is STILL OWED** — ruling 2, above.
|
||||
|
||||
v0.93.0 made the key survive. v0.94.0/v0.125.0/v0.195.0 make it come back. **Nothing yet restores a
|
||||
file with it.**
|
||||
**UPDATED 2026-08-05, after controller v0.198.0 + hub v0.95.0 (R-204):**
|
||||
|
||||
- **R-204 items 1–3 — CLOSED.** The reset code works without a restart; a re-issue no longer marks a
|
||||
healthy escrow stale (→ **R-196 CLOSED**); a unit restore states what it did NOT restore.
|
||||
- **R-204 item 4 — OPEN and unstarted:** a rebuilt box cannot obtain an off-site credential unaided.
|
||||
It needs an operator ruling on the one-shot credential design → **R-193**.
|
||||
- **Still open and untouched by this session, stated so nothing is presumed closed by association:**
|
||||
**R-202** (the orphan card's unconditional promise), **the ~1.2 GB orphaned-ciphertext deletion**
|
||||
(ruling 2 — still owed, still needs its own scoped session), and **R-198's retention, which remains
|
||||
UNIT-PROVEN ONLY** — nothing has superseded a key in production, and proving it needs a SECOND
|
||||
deliberate wipe. That retention drill is the next item, and it is not this session's.
|
||||
|
||||
v0.93.0 made the key survive. v0.94.0/v0.125.0/v0.195.0 make it come back. v0.197.0 got the file into
|
||||
the snapshot and the drill got it out again. **v0.198.0/v0.95.0 remove three of the four crutches the
|
||||
drill needed — the fourth is R-193, and until it goes the recovery is still operator-assisted.**
|
||||
|
||||
| ID | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|
|
||||
@@ -71,7 +85,7 @@ file with it.**
|
||||
| **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **CLOSED — SHIPPED 2026-08-04** (installer **1.25.0**; both live boxes corrected) | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking **THE GATE WAS RUN FIRST, AND IT MATTERED.** Before disabling anything, ep0 was read (read-only, Tier 2): prune jobs `prune-demo-felhom` and `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, `schedule 03:30`, `keep-last 2`, comment *"R-82 retention keep-last=2, server-side (box tokens are write-only)"* — and they have run **every day since 2026-07-27: 18 tasks, all `status=OK`**. The newest task log reads `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` / `keep ct/9201/2026-07-27…` / `keep ct/9201/2026-07-28…` / `TASK OK`. Retention happens, and it happens there. **A METHODOLOGICAL WARNING WORTH MORE THAN THE FIX.** Three separate queries said the OPPOSITE — *no prune jobs have ever run* — and **all three were broken instruments**: `worker-type` where the field is `worker_type`; the value `prune` where the worker type is `prunejob`; and `journalctl -u proxmox-backup` where the unit is `proxmox-backup-proxy`. A fourth reading (3 snapshots under keep-last 2) was mis-framed by CC and self-corrected — the third snapshot had landed AFTER that day's 03:30 window. Acting on any of them would have disabled the only pruning ATTEMPT while reporting that nothing prunes: a weekly false alarm traded for unbounded growth on the protected endpoint, invisible for months. **The gate is what caught it, and only because it demanded evidence rather than a verdict.** **Shipped:** installer **1.25.0** writes `keep_last: 0` on the offsite tier (the agent's existing guard `allowPBSPrune = !primary && keep_last > 0` already reads that as *never prune from the box* — no agent change), the justifying paragraph is rewritten to say where retention lives and cite R-89, and `hostinstall_gates.py` asserts it (red-proved: pinning `keep_last: 2` back fails the gate). **Both live boxes corrected in their own config** — `backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false` on demo-felhom and demo-hp, with the local tier untouched at `keep_last=3`. Served over HTTPS at `1.25.0` with `"keep_last":0` in the served bytes. **STILL TO OBSERVE:** the next weekly offsite run completing OK end-to-end. The change removes the failing step; the *schedule* proving it is next week's event, and this row should carry that line when it happens. | CC |
|
||||
| **R-194** | **PVE's permission cache delays every grant-state verdict by an unknown amount, so "the agent can read it" and "the ACL exists" are not the same measurement.** Observed twice while validating R-190's self-repair on demo-felhom 2026-08-04: both ACL rows for `/storage/felhom-backup` were deleted, and `GET /access/permissions` continued to report `Datastore.AllocateSpace` present — for **~40 s** in one run and **~16 minutes** in another. During that window the capability probe reads healthy and the self-repair does not fire | **OPEN** | — | **Why it matters beyond the delay:** it puts a floor under how fast a lost grant can be noticed, it makes any single permission read a lagging indicator, and — the interesting part — **it is a candidate contributor to R-190's own timeline**: a grant removed at an unknown moment could keep working until a cache expiry, which is exactly the shape of *worked at 04:44, refused at 09:24*. That does not explain what removed it, but it may explain when the refusal SURFACED, and the two have been treated as the same instant. **Not a defect in our code** — it is PVE behaviour, and the mitigation already tolerates it (the repair fires on the next probe after the cache clears). **What is worth deciding:** whether the store-grant probe should ALSO consult the storage content listing as a second signal, since that appeared to reflect the loss immediately (`{"data":[]}` while the permission read still said present) — two signals disagreeing is itself information, and today only one of them is read | CC |
|
||||
| **R-195** | ~~**A customer with no machine ever bound e-mailed an `expected_dbdump_missed` ERROR every morning.** `david` — a real prospective customer whose record was created 2026-08-01 16:51:49 with **hosts=0, host_deletions=0, host_reports=0, reports=0** — raised the alarm at 03:00 UTC on 08-02, 08-03 and 08-04~~ | **SHIPPED** (hub **v0.92.0**, 2026-08-04) | — | **The mechanism, and it is the interesting half: the skip that protects every other silent customer is keyed off having reported at least once.** `CheckBackupDeadlines`' down-skip reads `StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — a query **over the `reports` table**. A customer with zero reports appears in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. The skip therefore misses exactly the customer it would most obviously cover. (Corroborated live: `peti-felhom` is active with a host deleted 2026-07-15 and **does not** alarm — it has 482 old reports, so it is `down` and skipped.) The backup half was already safe (`reportJSON == ""` → skip); the DB-dump half had no guard at all. **Fix:** `store.HasEverBoundHost(customerID)` = a live `hosts` row **OR** a `host_deletions` tombstone, consulted once per customer at the top of the deadline loop. **The discriminator is deliberately "was a host EVER bound", NOT "has a report arrived"** — a box that was installed and went silent is a real fault and must keep alarming; that is the case the change could break and it has its own test. Fail-**open** on a read error (an unreadable binding must never SUPPRESS a real alarm), and the deferral is LOGGED with its own counter (the v0.73.0 Part-7 precedent: a quiet check must not look like a check that did not run). The anchored-verdict structure is untouched. **Red-proof observed:** deleting the guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` — verbatim the event `david` sent three mornings running. `david`'s record was NOT modified; the record was correct and the alarm was what was wrong | — |
|
||||
| **R-196** | **`escrow_stale` is wired to the ONE path that does not change the repo password, and absent from the path that does.** `ReissueCredentials` (`hub/internal/offsite/offsite.go:150-228`) resets **only** the Hetzner sub-account/box password and stages a fresh one-time secret — it contains no reference to a restic password and *cannot*, since that password is generated on the box and never leaves it except into the R-wrapped escrow. Yet it calls `MarkEscrowStale` on the stated grounds that *"the restic repo password just changed"* (`offsite.go:198-201`), and the same false premise is repeated at `api/handler.go:1067-1069` and in R-39's record | **OPEN — and it is now the blocker on the recovery path, not a documentation nit** | — | **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting a guarantee the code does not provide** — and the first where the comment factually describes a *different function*. It survived because its EFFECT (a stale escrow) is real, so nobody checked its CAUSE. **The live defect, not a documentation nit:** on the ordinary Re-issue shape (a consumed-but-failed install on a box that still holds its `repo_password` file) the box re-applies, `WriteOffboxSecrets` finds the file present and **keeps it**, the repo password is unchanged — and the hub has told the customer in Hungarian that their recovery escrow is stale and asked them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** **The inverse is the worse half and is R-193's:** demo-felhom's repo password *did* change on 2026-08-03 with **no Re-issue anywhere in its history** (its only `escrow_stale`/`offsite_reissued` pair is dated 2026-07-21 08:29:29) and therefore **nothing marked its escrow stale for 13 h**. **Fix shape:** mark the escrow stale on the evidence that it IS stale — a changed `restic_pw_sha256` (→ R-197) — not on a Re-issue; and correct all three comments in the same commit. **Not established, so not asserted:** whether the 2026-07-21 Re-issue on demo-felhom re-sealed an unchanged password (the predicted false-staleness shape) — `host_escrow_superseded` holds only two rows in the whole DB, both from 2026-08-04, so the prior generation is not retained. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q4 **FIVE instances, not three** — the spec expected three and named two; a census found five: `offsite/offsite.go` (the `MarkEscrowStale` justification), `api/handler.go` (the F3 re-enroll comment) and **three in `store/store.go`** (the `stale_at` ALTER comment, the `MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). All five now state what the code does, name the correction, and cite the recon; the staleness mark is documented as **precautionary** (the box's re-apply may mint a fresh repository password — the guest-rebuild shape) rather than evidential, with R-197's measured signal named as the evidential one. **THE BEHAVIOUR IS UNCHANGED AND THIS ROW STAYS OPEN:** on the ordinary re-issue shape — a box that still holds its `repo_password` file — the password does not change and the hub still marks a healthy escrow stale and asks the customer for an unnecessary ceremony. That is a behaviour change and must not ride a comment-correction release; it is also now **more** consequential than when filed, because under R-198 an unnecessary ceremony is no longer harmless bookkeeping — it supersedes a blob. Fix shape unchanged: mark stale on the evidence that it IS stale (R-197's hash comparison), not on a re-issue. **MEASURED LIVE 2026-08-04 night, in the flow where it does real damage.** After the Re-issue a rebuilt box needs to configure its off-site tier, `stale_at` was set at `2026-08-04 20:15:49` **while `restic_pw_sha256` was unchanged** — the escrow perfectly covered the box's current (recovered) key. The consequence is not cosmetic: a stale escrow makes the hub withhold the hash from the ACK, `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` refuses every off-site run. **The only documented way out is a fresh ceremony, which supersedes the identity blob and destroys the key being recovered.** This row's fix — mark the escrow stale on the evidence that it IS stale (a changed `restic_pw_sha256`, R-197's comparison), not on a Re-issue — is now on the critical path for R-201/R-204, not a tidy-up. Full chain: `audits/DRILL-r201-night-run-2026-08-04.md` §3(b) | CC |
|
||||
| **R-196** | **`escrow_stale` is wired to the ONE path that does not change the repo password, and absent from the path that does.** `ReissueCredentials` (`hub/internal/offsite/offsite.go:150-228`) resets **only** the Hetzner sub-account/box password and stages a fresh one-time secret — it contains no reference to a restic password and *cannot*, since that password is generated on the box and never leaves it except into the R-wrapped escrow. Yet it calls `MarkEscrowStale` on the stated grounds that *"the restic repo password just changed"* (`offsite.go:198-201`), and the same false premise is repeated at `api/handler.go:1067-1069` and in R-39's record | **CLOSED 2026-08-05 — hub v0.95.0** | — | **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting a guarantee the code does not provide** — and the first where the comment factually describes a *different function*. It survived because its EFFECT (a stale escrow) is real, so nobody checked its CAUSE. **The live defect, not a documentation nit:** on the ordinary Re-issue shape (a consumed-but-failed install on a box that still holds its `repo_password` file) the box re-applies, `WriteOffboxSecrets` finds the file present and **keeps it**, the repo password is unchanged — and the hub has told the customer in Hungarian that their recovery escrow is stale and asked them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** **The inverse is the worse half and is R-193's:** demo-felhom's repo password *did* change on 2026-08-03 with **no Re-issue anywhere in its history** (its only `escrow_stale`/`offsite_reissued` pair is dated 2026-07-21 08:29:29) and therefore **nothing marked its escrow stale for 13 h**. **Fix shape:** mark the escrow stale on the evidence that it IS stale — a changed `restic_pw_sha256` (→ R-197) — not on a Re-issue; and correct all three comments in the same commit. **Not established, so not asserted:** whether the 2026-07-21 Re-issue on demo-felhom re-sealed an unchanged password (the predicted false-staleness shape) — `host_escrow_superseded` holds only two rows in the whole DB, both from 2026-08-04, so the prior generation is not retained. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q4 **FIVE instances, not three** — the spec expected three and named two; a census found five: `offsite/offsite.go` (the `MarkEscrowStale` justification), `api/handler.go` (the F3 re-enroll comment) and **three in `store/store.go`** (the `stale_at` ALTER comment, the `MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). All five now state what the code does, name the correction, and cite the recon; the staleness mark is documented as **precautionary** (the box's re-apply may mint a fresh repository password — the guest-rebuild shape) rather than evidential, with R-197's measured signal named as the evidential one. **THE BEHAVIOUR IS UNCHANGED AND THIS ROW STAYS OPEN:** on the ordinary re-issue shape — a box that still holds its `repo_password` file — the password does not change and the hub still marks a healthy escrow stale and asks the customer for an unnecessary ceremony. That is a behaviour change and must not ride a comment-correction release; it is also now **more** consequential than when filed, because under R-198 an unnecessary ceremony is no longer harmless bookkeeping — it supersedes a blob. Fix shape unchanged: mark stale on the evidence that it IS stale (R-197's hash comparison), not on a re-issue. **MEASURED LIVE 2026-08-04 night, in the flow where it does real damage.** After the Re-issue a rebuilt box needs to configure its off-site tier, `stale_at` was set at `2026-08-04 20:15:49` **while `restic_pw_sha256` was unchanged** — the escrow perfectly covered the box's current (recovered) key. The consequence is not cosmetic: a stale escrow makes the hub withhold the hash from the ACK, `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` refuses every off-site run. **The only documented way out is a fresh ceremony, which supersedes the identity blob and destroys the key being recovered.** This row's fix — mark the escrow stale on the evidence that it IS stale (a changed `restic_pw_sha256`, R-197's comparison), not on a Re-issue — is now on the critical path for R-201/R-204, not a tidy-up. Full chain: `audits/DRILL-r201-night-run-2026-08-04.md` §3(b) | CC **CLOSED (hub v0.95.0, 2026-08-05).** The pessimistic `MarkEscrowStale` and its `escrow_stale` event are GONE from `ReissueCredentials`; `offsite_reissued` is untouched. The removal was not a softening — nothing now guesses at re-issue time. **The case the mark guessed at is measured in two places, and one of them the mark was BLINDING:** a stale flag empties `restic_pw_sha256` from the report ACK, which is the exact value the controller's Scenario-F re-check (`report.EscrowAutoConfirmer.reconcileEscrowed`) compares against its live repo password every cycle — so oLine truncated
|
||||
| **R-197** | **The hub holds both halves of the evidence that a box's offsite DATA key changed, and reads neither.** `restic_pw_sha256` is stored on `host_escrow` and carried to `host_escrow_superseded` on every re-escrow. Comparing the two is what let this spike answer its hardest question in one query — and **nothing in the hub does it** | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **Why this is the cheapest real fix on the table.** A changed repo password means the previous offsite repository is now unopenable by the box, i.e. the customer's off-site history is orphaned. That is the single most consequential state change in the backup system, it is **already fully observable from data the hub owns**, and today it produces **no event, no e-mail, no card and no log line** — demo-felhom's went unremarked for 13 hours and would have gone unremarked indefinitely had this spike not run. **Two-line verdict:** on `SaveHostEscrow`, if the incoming `restic_pw_sha256` differs from the row being superseded, emit a distinct operator event naming the orphaned generation. **Deliberately an EVENT, not a heal** — nothing should act on this automatically until R-193's (c)-vs-accept decision is taken; the point is that the operator learns on the day. **Pair with R-196**, which is the same signal aimed at the right trigger. **Generalises past this row:** *a comparison the system could be making from data it already stores, and is not, is a silence with no cost of entry* — cf. R-190's store-grant probe, where the state was read and the TRANSITION was not. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q8 option (d) **SHIPPED.** `SaveHostEscrow` returns the hash it replaced; `handleHostEscrowPut` raises **`offsite_repo_key_changed`** when both hashes are known and differ. **Edge-triggered** (once per supersession, never per report — the dispatcher owns cooldown), **operator-only** (registered in `notify.operatorOnlyEvents` in the same commit that mints the type, because a missing `customerMessages` entry is NOT a block — the v0.78.0 defect), and **no hash value travels** in the message or the details. The in-between shapes (a first-ever hash, a hash-less supersession) are LOGGED rather than dropped, so *"we chose not to alarm"* and *"the check did not run"* never look identical. **Severity = warning, chosen for the world v0.93.0 creates:** before R-198 a changed key meant the previous history was unopenable by anyone ever, which would have argued for `error`; from v0.93.0 the superseding ceremony retains the old identity blob, so the fact is *"this customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss. `warning` also routes (the dispatcher treats `info` as an intentional non-notify). Driven through the real endpoint in test, not by calling the emitter. **Red-proof observed:** removing the comparison from the escrow PUT → the changed-key scenario fails with *"the repository key demonstrably changed and NO signal was raised"* while the unchanged-key scenario still passes. | CC |
|
||||
| **R-198** | **The hub's superseded-escrow retention does NOT retain the offsite repository password — and the ceremony the system tells the customer to run is what destroys the last copy.** `host_escrow_superseded` has **no `identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) copies only `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256`. `blob` is the **K-escrow** (the PBS datastore key, PBS-native scrypt); the **restic repo password lives in `identity_blob`** (`felhom-agent/internal/escrow/identity.go:34-39`, age-wrapped `IdentityBundle`). Measured live: both hosts' current rows hold `blob`=383 B **and** `identity_blob`=572 B; both superseded rows hold `blob`=383 B and nothing else | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **This is the NINTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does not provide, and the first that is ALSO customer-facing copy.** The claim appears three times: the schema comment (`store.go:370-375`, *"so the old passphrase stays customer-R-recoverable … turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable'"*), the capability map's escrow row, and — in Hungarian, to the customer, on the orphan card — `controller/internal/web/templates/backups_remote.html:66,69` (*„a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók lehetnek"*). **For the offsite restic repository, the incident it names, all three are false.** **Why it is worse than a missing column:** a rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`, `offbox.go:570`), the card says „Helyreállítási kód szükséges", and the controller's own detector logs *"run the escrow ceremony"* (`report/escrow_confirm.go:100`) — so **the prescribed remedy is the act that overwrites `host_escrow.identity_blob` and loses the old password forever**. Both demo boxes crossed that line on **2026-08-04 at 07:15:36 (demo-hp) and 07:20:08 (demo-felhom)**. **This is an independent, stronger reason the 51 orphaned demo snapshots are unrecoverable than "nobody kept the recovery codes" — keeping R would not have helped.** **Fix shape (small):** add `identity_blob` to the superseded table and to `demoteCurrentEscrowTx`'s SELECT list; pin it with a test that asserts the CONSEQUENCE (a superseded row can still yield a repo password) rather than the mechanism. **Then correct all three claims in the same commit** — including the Hungarian card, which must not promise what the system cannot do. **Prerequisite for R-193's (d)-alone branch** and for the drill. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §7 **SHIPPED.** `host_escrow_superseded` gains `identity_blob` (CREATE + additive `ALTER TABLE`) and `demoteCurrentEscrowTx` carries it, so **both** callers — re-escrow and host-delete demotion — are fixed by one change to the routine its own comment calls *"THE ONE escrow row-copy routine"*. `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob is reachable from Go at all. The table comment now records that the ruling stated above it was not met and what that cost. **Tests assert the CONSEQUENCE, which is why the existing one stayed green:** `TestSaveHostEscrow_RetainsSuperseded` asserted that a retained row exists carrying the old K-blob and passed throughout; `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still yield a repository password, and pins the load-bearing ordering (the identity blob is written by `SaveHostDRBundle` AFTER `SaveHostEscrow`, so the demote sees the PREVIOUS generation — if that inverts, the retained bytes would be the new blob filed under the old hash, recoverable-looking and wrong). `TestDeleteHost_DemotesIdentityBlob` proves the shared routine through its other caller. **Red-proofs, both observed failing:** dropping `identity_blob` from the copy (production behaviour ≤ v0.92.0) fails BOTH scenarios; fixing only the re-escrow caller fails the delete scenario while the re-escrow one passes — the §8.2 mistake, demonstrated rather than asserted. **Nothing was backfillable and it was CHECKED, not deduced:** rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; the live database holds exactly 2 retained rows, both from 2026-08-04, both `identity_blob` NULL. **Who the fix protects, measured live:** 2 of 2 hosts with a current escrow carry an identity blob (`demo-felhom-8363b5`, `demo-hp-bb76ea`) — their NEXT ceremony now retains a recoverable off-site key instead of destroying one. **STILL UNIT-PROVEN ONLY after the 2026-08-04 night drill.** Part 2 (wipe again, do NOT recover, let a ceremony seal a DIFFERENT password, then inspect the superseded row's `identity_blob`) was gated on the first drill passing and **did not run** — the verdict was not reached, and a second wipe wouldLine truncated
|
||||
| **R-199** | **The hub serves recovery blobs on two endpoints that have no client anywhere in the system.** `handleReEnroll` (`hub/internal/api/dr.go:101`) and `handleGetRestoreDirective` (`:155`) return `identity_escrow_b64` + `k_escrow_b64`, gated on operator-armed recovery mode. Census: **zero** callers in `felhom-agent` (no `ReEnroll` symbol at all; `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`), **zero** in the hub UI or any template, **zero** in `scripts/` or any runbook | **SHIPPED + PROVEN-LIVE 2026-08-04** (hub **v0.94.0**, agent **v0.125.0**) | — | **The documented retrieval path is a human with `sqlite3`:** `SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …` on a `kubectl cp`-ed `hub.db` — recorded in project memory as the 2026-07-04 S5 prep, where hub-side blob serving was explicitly deemed *"Part 3 NOT needed"*. **This is the built-but-never-wired class at the DR capstone**, and it is why the chain from a dead node to an open repository has no automatable middle. **What a recovery flow actually needs is smaller than what exists:** a narrow `GET /hosts/<id>/escrow` authed with the box's own per-host key, serving opaque bytes to the box that owns them — zero-knowledge untouched, and far lighter than `re-enroll`, which **rotates the host API key** and returns the new key in the response body (`dr.go:130,148`). **Decide before building:** whether `re-enroll`/`restore-directive` should get a client, be replaced by the narrow GET, or be retired. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 6 **LINKS 6, 7 AND 8 ARE ASSEMBLED AND WALKED.** **Link 6:** `GET /api/v1/hosts/{host_id}/escrow` — the box-authenticated MIRROR of the PUT that stores the blob, self-scoped by the per-host key (global may read any, the same asymmetry the PUT has). The operator-driven DR endpoints are UNTOUCHED and pinned by a test that exercises them with recovery mode off and on. **Link 7:** `POST /escrow/recover-offsite-password` on the agent's pinned local API gives `UnwrapIdentityBundle` its first production caller in two months. **Link 8:** it extracts and returns **only** the repository password (not the tunnel token, not the PBS token, not the WG key — the controller is a trust tier down). **PROVEN ON HARDWARE, demo-felhom, 2026-08-04 13:49 CEST:** on-disk `c60c8bc737a6…` vs recovered `c60c8bc737a6…` — **MATCH**, and the same hash the hub independently stores as `restic_pw_sha256`, so three sources agree. **Scenario B proven live 5 minutes earlier** with a deliberately wrong code: hub served the blob (572 B, `self_scope=true`), agent logged *the recovery code did not unwrap the identity escrow … exit status 1*, nothing written — which also proves links 6 and 7 ran independently of the success. **Scenario E proven live:** both retrievals raised `escrow_blob_served` (warning, operator-only); the first mailed the operator, the second was cooldown-suppressed AND that suppression is itself recorded; the customer leg reads `skipped/operator_only` on both. **R persisted nowhere, searched not claimed:** 0 lines in the agent journal, 0 in the controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover `felhom-idesc-*` staging dirs, and the staged-secret dir empty — with a **positive control** (a planted copy was found: 1, then 0 after removal) so the sweep is a measurement rather than an unfalsifiable absence. **THE §8.2 TRADE, made deliberately and recorded in the handler:** obtaining the blob used to require the operator to arm recovery mode; it now needs only the box's own credential. They still cannot open it (the hub never held R; a wrong code fails closed at age's scrypt KDF). `escrowSelfServiceRetrieval` is a single named constant — flipping it to false re-imposes recovery mode and changes nothing else, so the operator can overrule the trade for the cost of a boolean. **Red-proofs observed:** removing the ownership check served host B's blob to host A; removing the audit record made the retrieval silent; returning `PBSToken` instead of `ResticRepoPassword` yielded a plausible bundle with a non-matching key; commenting the `Options.EscrowRecovery` wiring failed the AST seam test. **Seam discipline:** the wiring is asserted by walking `main` → `runDaemon` → `buildLocalAPIServer` and checking the composite literal, not by `strings.Contains` — links 6 and 7 were two of this project's six built-but-never-wired instances and the fix must not become the seventh | CC |
|
||||
@@ -79,7 +93,7 @@ file with it.**
|
||||
| **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **PASSED + PROVEN-LIVE 2026-08-04** — the customer file came back byte-identical | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there **MATERIALLY ADVANCED 2026-08-04, AND THE DISTINCTION MATTERS.** What was proven on hardware is that the offsite repository **password** comes back out of the sealed bundle byte-identical (R-199). What has STILL never happened: a recovered password **installed**, an existing repository **reopened** under one, and a **file restored** from it. **The drill's pass condition is unchanged and is not this** — it is a byte-identical sha256 of a sentinel file after a wipe and reinstall, and "the store opened" was explicitly ruled insufficient. Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10. **The drill is now cheaper and better-founded than when it was designed:** links 6–8 are walked, so a failure during the drill can be localised instead of being an undifferentiated "recovery did not work", and R-198 means the ceremony that creates the drill's recovery code no longer destroys the key it is meant to protect **THE DRILL RAN AND DID NOT REACH ITS VERDICT, and stopping was the correct call** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). Steps 1–4 completed; the wipe never happened; **nothing irreversible was done**. It halted because **the sentinel file was not in the off-site snapshot** (R-203) — wiping would have destroyed the only copy and proven nothing. **Sentinel sha256 `643166269103a25c…`, still on the box.** **What the attempt established live, all of it new:** (1) a rebuilt box's off-site run **refuses** with the orphan card and pushes `offbox_repo_orphaned` — the spike's predicted third outcome, measured for the first time, and it does NOT silently start a fresh history (this closes R-193's Q3); (2) the **orphan reset works** — move-aside to `/home/felhom-repo.orphaned-20260804`, never delete, fresh repo initialised, `offbox_repo_reset` pushed (operator-authorised during the session); (3) demo-hp's pre-rebuild history is **permanently unrecoverable** — its key sits in superseded row id 3 with `identity_blob` NULL, superseded 07:15:36, **four hours before v0.93.0 fixed the retention**; (4) **a precondition the runbook did not contain:** neither pre-existing off-site-toggled app has a restorable file leg — both are named-volume-only, which the off-site tier tars but the customer restore never unpacks — so a file-leg app (`calibre-web`, mandatory `userdata: media/books`) had to be deployed, and it is now in place as the fixture. **TO RESUME:** fix or scope R-203, re-run steps 4–5 and confirm the sentinel IS in the snapshot **by listing it, not by a green status**, then P4 + the STOP + steps 6–11. Everything else is already staged: versions, recovery code, working repository, file-leg app, sentinel. **The pass condition is unchanged** — a byte-identical sentinel sha256, not "the store opened" **UNBLOCKED 2026-08-04 by R-203 (controller v0.197.0).** The sentinel now lands in the off-site snapshot and is listed there by name and size — the thing whose absence halted the drill. **Everything the resumed run needs is already in place on demo-hp:** agent v0.125.0, controller v0.197.0, the recovery code held by the operator (`R_DEMO-HP`), a working off-site repository (3 snapshots), `calibre-web` deployed with a mandatory userdata path, and the sentinel at sha256 `643Line truncated
|
||||
| **R-202** | **The orphan card promises the customer their old backups "may later be restorable with the matching recovery code" — and after v0.93.0 that is true for supersessions from now on and FALSE for anything already orphaned.** `controller/internal/web/templates/backups_remote.html:66,69` states it unconditionally, in Hungarian, on the one surface where being wrong costs most | **OPEN — Part 5 hit its gate 2026-08-04; the card is UNTOUCHED and the sentence is still live** | R-199/R-201 (which generation an orphaned repo belongs to is not knowable to the box today) | — | **THE GATE, and why it was hit rather than squeezed past.** The condition was: ship it iff the hub can tell a box what it needs with **one** additional boolean on the escrow ACK it already sends. The hub *can* cheaply compute *"≥1 retained blob for this host carries an identity blob"* — one correlated predicate in `GetEscrowStatusForCustomer`, and the controller even has the right seam already (`SetEscrowStale`/`StaleBlob` is exactly this shape). **But that boolean does not answer the card's question.** The card renders on `RepoState == "orphaned"`, and the promise is about *the key THIS orphaned repository was written under*. A box does not know which escrow generation the orphaned remote belongs to; a box with a pre-v0.93.0 orphan and a post-v0.93.0 supersession would read the boolean TRUE and the promise would still be false — a **conditional** falsehood that looks verified, which is strictly worse on a customer-facing card than today's hedged one. **What would actually make it truthful** is knowing the orphaned repo's generation, which is the same knowledge R-199/R-201's unassembled chain needs. **Interim exposure, stated rather than buried:** the sentence remains live and remains false for both demo boxes. **Cheapest honest interim** (not taken here — it is a customer-copy change and the gate said leave it alone): drop the recoverability clause and say only that the old history is set aside and not deleted, which is true unconditionally | CC + operator |
|
||||
| **R-203** | **A customer-declared MANDATORY data directory was silently absent from the off-site snapshot while the run reported `ok`.** Measured live on demo-hp 2026-08-04: `calibre-web` declares `userdata: media/books class: mandatory`; its live bind is `/mnt/sys_drive/userdata/media/books` (the sentinel file was there), while the off-site capture set looked for `/mnt/sys_drive/felhom-data/userdata/media/books`, which does not exist. Result: `[WARN] mandatory data path missing on disk, skipped from offsite`, then `backed up calibre-web (…, 0 mandatory path(s))` and `backup OK: 3 app(s), 3 snapshot(s)` — `last_status: ok`, `last_success` stamped, nothing customer-visible, nothing hub-visible | **SHIPPED + PROVEN-LIVE 2026-08-04** (controller **v0.197.0**) | — | **THE MECHANISM, from source.** `NamespaceRoot(drivePath, inGuestDrive)` (`appbackup/paths.go:28-33`) appends the `felhom-data` segment **when the drive IS the system data path** (`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`, `backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is `<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an app deployed at `HDD_PATH=/mnt/sys_drive`, the two produce different directories. **The same compose used BOTH roots**, from one deploy: `${IMPORT_PATH}` → `/mnt/sys_drive/felhom-data/userdata/import/calibre` (with the segment), `${USERDATA_PATH}` → `/mnt/sys_drive/userdata/media/books` (without). **WHAT IS MEASURED vs NOT, because it changes the fix.** MEASURED: the paths disagree, the mandatory directory is absent from the snapshot, the run says `ok`, and the only signal is a container-log WARN. **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a SUPPORTED choice — it was used because demo-hp's only registered drive (`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working as designed), while `/mnt/sys_drive` was **accepted** (HTTP 202). **EITHER BRANCH IS A DEFECT:** if the system drive is a supported app namespace, userdata resolution is wrong for every app on it and their mandatory directories are silently unprotected; if it is not supported, the deploy accepted a namespace it should have refused one call after refusing the NAS. **NOT a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths (everything of theirs is in named volumes), so they are unaffected and their snapshots are real. **Fix shape:** make the two roots one function, whichever is right — and make a skipped MANDATORY path a customer/hub-visible signal rather than a WARN, because `ok` with a missing mandatory directory is this project's own *a path the customer thinks is protected is not in the snapshot* shape. Source: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 **BOTH HALVES SHIPPED.** **(1) The paths.** `appbackup`'s helpers take a NAMESPACE ROOT; the census found **FIVE** bare-drive-path callers, not the four the spec named — the fifth is the **FileBrowser mount builder** (`web/handlers.go`), i.e. the customer's own file browser would have shown the wrong directory on a non-enrolled path (latent: the system drive is deliberately never a registered `StoragePath`). The rule now has **ONE expression** (`appbackup.NamespaceRootFor` / `IsEnrolledDrive`); there were already **two** copies and **they differed** — `backup.Manager.namespaceRoot` compared without `filepath.Clean`, `stacks.Manager.inGuest` with it, so a trailing slash from config would have flipped the mode in one package and not the other. `ComputeFabBuckets` now receives the namespace root, which is what `ComputeCaptureSet` has always received, so the export and the backup describe the same directories by construction. **(2) The verdict.** `last_status` gains **`incomplete`** — minted, because `ok`|`error`|`running` had nothing meaning *"it ran, and this app is not fully protected"*. **Not `error`:** the rest of the run worked, so `SnapshotCount` and the `LastSuccess` anchor still record what WAS captured. It reaches the operator through the **existing** per-run digest (`backup_run_failures`) — a new event type would be a two-repo change and the hub drops anything outside `allowedEventTypes`. **§8.4's narrowing is a NO-OP and no customer warning disappears:** `TierOffsite`'s `tierKeeps()` already admits mandatory only, demonstrated by widening the tier filter alone and watching the class check hold the line. **THE SPEC'S §8.3 RISK DOES NOT EXIST, and this is the correction owed:** `ExportDataMounts` lives in `delete.go` but is **export-only** — its single production caller is the `.fab` adapter, nothing deletes on its result, and the delete path's own guard `ProtectedHDDPaths` is layout-agnostic by construction (it protects BOTH `<hdd>/…` and `<hdd>/felhom-data/…`). It shipped as its own commit anyway. **PROVEN LIVE on demo-hp:** the bind moved `/mnt/sys_drive/userdata/Line truncated
|
||||
| **R-204** | **A rebuilt box can recover its off-site key and still cannot use it: the remedy that reconfigures the tier is the thing that blocks the recovery.** Measured end to end on demo-hp during the 2026-08-04 night drill, after a real controller-data wipe | **OPEN — the four steps between a recovered key and a restored file** | — | **THE CHAIN, each link measured.** (1) A rebuilt controller **cannot configure its off-site target at all**: `offsite-apply: consume one-time password: no unconsumed offsite password` — the previous controller consumed it (ledger: created `07:11:51`, consumed `07:12:06`). That is R-193, reconfirmed live. (2) The documented remedy is an operator **Re-issue**, which works — and **sets `stale_at` on the escrow** (measured: `2026-08-04 20:15:49`) **while `restic_pw_sha256` is unchanged**, i.e. R-196's false staleness. (3) A stale escrow makes the hub **withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses every off-site run**. (4) The only documented way to clear a stale escrow is **a fresh ceremony — which supersedes the identity blob and destroys the key being recovered.** **So the recovery and its precondition are mutually exclusive as built.** The key comes back (proven — `8a9e33aa4da6…` recovered byte-identical after the wipe, and installed) and then cannot be used to open the repository. **(5) A FOURTH link, undocumented anywhere:** a rebuilt box is **unclaimed**, and the claim gate correctly intercepts every non-claim route (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`), so **no controller endpoint responds at all** until the customer re-claims. That step appears in no design document and comes first. **Fix shape, not decided here:** either the Re-issue stops marking the escrow stale on evidence it does not have (R-196's own fix), or a recovered-and-verified key is allowed to confirm the escrow without a ceremony — the hash comparison that `EscrowAutoConfirmer` already performs is exactly the evidence needed, and it is being withheld precisely when it would be conclusive. **Do NOT fix by widening `OffboxRunnable`** — the atomicity guarantee it enforces (no un-recoverable ciphertext) is the reason the escrow exists. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3 **ALL FOUR WALKED AND MEASURED during the passing 2026-08-04 drill, so this row is now a known-good manual runbook AND the gap list.** (1) **R-193:** a rebuilt controller cannot configure its off-site tier — `no unconsumed offsite password` (ledger: created `07:11:51`, consumed `07:12:06` by its predecessor). Remedy: operator Re-issue. (2) **The claim gate:** a rebuilt box is unclaimed, so the gate intercepts EVERY controller endpoint (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`) — a first step of every recovery that appears in no design document. **And the local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to `settings.json` while the RUNNING controller keeps its old copy in memory, so `effectiveClaimCode()` never sees it and the claim fails with *"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming** — two attempts failed before this was diagnosed. (3) **R-196:** the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged → the hub withholds the hash → auto-confirm can never fire → `OffboxRunnable` refuses every run. Cleared here with the **manual** confirm (`/backup/offbox/confirm-escrow`) — NOT a ceremony, which would have superseded the identity blob and destroyed the recovered key. (4) **`mode=unit` is the restore default and returns the recovery unit, NOT the userdata leg.** A customer told to "restore from off-site" gets their app definition back and not their documents, and nothing in that outcome says so. **Fix priorities, in the order they hurt:** (4) is a customer-facing trap on the last step; (3) is R-196's fix; (2) needs the escape hatch to reload settings (or the claim state to survive a rebuild); (1) is R-193's restage. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3–§4 | CC + operator |
|
||||
| **R-204** | **A rebuilt box can recover its off-site key and still cannot use it: the remedy that reconfigures the tier is the thing that blocks the recovery.** Measured end to end on demo-hp during the 2026-08-04 night drill, after a real controller-data wipe | **ITEMS 1–3 CLOSED 2026-08-05; ITEM 4 OPEN (→ R-193)** | — | **THE CHAIN, each link measured.** (1) A rebuilt controller **cannot configure its off-site target at all**: `offsite-apply: consume one-time password: no unconsumed offsite password` — the previous controller consumed it (ledger: created `07:11:51`, consumed `07:12:06`). That is R-193, reconfirmed live. (2) The documented remedy is an operator **Re-issue**, which works — and **sets `stale_at` on the escrow** (measured: `2026-08-04 20:15:49`) **while `restic_pw_sha256` is unchanged**, i.e. R-196's false staleness. (3) A stale escrow makes the hub **withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses every off-site run**. (4) The only documented way to clear a stale escrow is **a fresh ceremony — which supersedes the identity blob and destroys the key being recovered.** **So the recovery and its precondition are mutually exclusive as built.** The key comes back (proven — `8a9e33aa4da6…` recovered byte-identical after the wipe, and installed) and then cannot be used to open the repository. **(5) A FOURTH link, undocumented anywhere:** a rebuilt box is **unclaimed**, and the claim gate correctly intercepts every non-claim route (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`), so **no controller endpoint responds at all** until the customer re-claims. That step appears in no design document and comes first. **Fix shape, not decided here:** either the Re-issue stops marking the escrow stale on evidence it does not have (R-196's own fix), or a recovered-and-verified key is allowed to confirm the escrow without a ceremony — the hash comparison that `EscrowAutoConfirmer` already performs is exactly the evidence needed, and it is being withheld precisely when it would be conclusive. **Do NOT fix by widening `OffboxRunnable`** — the atomicity guarantee it enforces (no un-recoverable ciphertext) is the reason the escrow exists. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3 **ALL FOUR WALKED AND MEASURED during the passing 2026-08-04 drill, so this row is now a known-good manual runbook AND the gap list.** (1) **R-193:** a rebuilt controller cannot configure its off-site tier — `no unconsumed offsite password` (ledger: created `07:11:51`, consumed `07:12:06` by its predecessor). Remedy: operator Re-issue. (2) **The claim gate:** a rebuilt box is unclaimed, so the gate intercepts EVERY controller endpoint (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`) — a first step of every recovery that appears in no design document. **And the local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to `settings.json` while the RUNNING controller keeps its old copy in memory, so `effectiveClaimCode()` never sees it and the claim fails with *"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming** — two attempts failed before this was diagnosed. (3) **R-196:** the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged → the hub withholds the hash → auto-confirm can never fire → `OffboxRunnable` refuses every run. Cleared here with the **manual** confirm (`/backup/offbox/confirm-escrow`) — NOT a ceremony, which would have superseded the identity blob and destroyed the recovered key. (4) **`mode=unit` is the restore default and returns the recovery unit, NOT the userdata leg.** A customer told to "restore from off-site" gets their app definition back and not their documents, and nothing in that outcome says so. **Fix priorities, in the order they hurt:** (4) is a customer-facing trap on the last step; (3) is R-196's fix; (2) needs the escape hatch to reload settings (or the claim state to survive a rebuild); (1) is R-193's restage. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3–§4 | CC + operator **OUTCOME 2026-08-05 — controller v0.198.0 + hub v0.95.0.** **Item 1 (the reset code needs a restart) — CLOSED, proven live.** `effectiveClaimCode` reads through to the persisted claim state, so a code minted by the separate `--print-reset-code` process is seen without a restart; the precedence rule between settings and config is unchanged. Read-through, not a TTL: a TTL leaves a window in which a superseded code still works, and that is the mutation `TestClaimCode_SupersededByASecondMint_RefusedImmediately` kills. Fails closed on an unreadable state. **Live on demo-felhom 9201, nothing restarted (`restarts=0`, container older than both mints): the superseded code returned „Hibás vagy lejárt kód" and the currenLine truncated
|
||||
| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **HALF SHIPPED** (hub **v0.93.0**) — **the guard's SCOPING stays OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** unLine truncated
|
||||
| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** — the (c) decision is TAKEN (accept the risk); the chain is still unassembled | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/esLine truncated
|
||||
| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
|
||||
|
||||
@@ -59,14 +59,14 @@
|
||||
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
|
||||
| R-198 | **The superseded-escrow retention kept the K-escrow and dropped the identity blob** — so it preserved the PBS key and destroyed the offsite restic repository password, with the escrow CEREMONY as the destroying act | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | Corrects the escrow-ceremony capability-map row's v0.60.0 retention claim (was struck through 2026-08-04, now restated as what ships). One column + one SELECT list, in the shared `demoteCurrentEscrowTx`, so both callers are fixed at once. Nothing backfillable. **Load-bearing, not tidy:** with R-193(c) refused, customer-present recovery is the only route back from a rebuild and it runs entirely through the retained identity blob |
|
||||
| R-197 | **The hub stored both halves of "did this box's offsite data key change" and compared them nowhere** | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | New `offsite_repo_key_changed` (warning, operator-only, edge-triggered, no hash values). Flips nothing in the capability map — it adds a signal, not a capability |
|
||||
| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **comments corrected (hub v0.93.0); the BEHAVIOUR stays open** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping |
|
||||
| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **CLOSED — behaviour shipped hub v0.95.0 (2026-08-05)** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping **Collapsed 2026-08-05.** The comment correction landed in v0.93.0; the BEHAVIOUR landed in v0.95.0 with R-204 item 2 — the precautionary `MarkEscrowStale` and its `escrow_stale` event are gone from `ReissueCredentials`. Removing a signal was safe because the real case is measured twice over, and the mark was **blinding** one of them: a stale flag empties `restic_pw_sha256` from the ACK, which is what the controller's per-ACK re-check compares against. |
|
||||
| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
|
||||
| R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 6–8 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
|
||||
| R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
|
||||
| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
|
||||
| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
|
||||
| R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
|
||||
| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **OPEN — the wall the 2026-08-04 night drill hit** | Blocks R-201's verdict. Fix is most likely R-196's (mark stale on evidence, not on a Re-issue) or letting a verified recovered key confirm the escrow without a ceremony |
|
||||
| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **ITEMS 1–3 SHIPPED (controller v0.198.0 + hub v0.95.0, 2026-08-05); ITEM 4 OPEN → R-193** | Collapsed per the coupling rule. The fix taken was R-196's — remove the guess, keep the two MEASURED signals (the controller's per-ACK hash re-check, which the stale flag was blinding, and R-197's supersession detector). Plus the claim-code read-through and the restore-scope wording. **Flips the capability-map recovery row from four crutches to one**, and that one is R-193's operator Re-issue, which keeps the row's *with a person present* qualifier. Does NOT touch R-198's retention, which is still unit-proven and needs the second drill |
|
||||
| R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet |
|
||||
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
|
||||
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
||||
|
||||
Reference in new issue
Block a user