docs: R-204 ALL FOUR items closed; R-193 credential half; R-192 by replacement; R-212 filed
gates / gates (push) Successful in 8s

- OPEN-ITEMS: R-204 all four CLOSED with both 2026-08-05 rulings recorded (the
  declared-state trigger and its four-meanings-of-absence reasoning; the recovery
  preview's dashboard-password exposure accepted as metadata, not content).
  R-193's credential half CLOSED, screen + deletion still open. R-192 CLOSED by
  REPLACEMENT. R-202 untouched.
- R-212 MINTED (R-211 was the highest, grepped): the orphaned-ciphertext deletion
  HALTED at its STOP because the measured paths do not match the register — three
  set-aside stores totalling ~1.45 GB, and the thing that is exactly 1.2 GB is
  demo-felhom's LIVE repo. Nothing was deleted.
- capability map: all four interventions closed; the row KEEPS a qualifier for a
  new reason — no step needs an operator, but there is no customer-facing recovery
  screen, and the journey has not been re-walked end to end.
- 07-backup-architecture 7.0: the four-step table updated; the declaration-vs-
  inference reasoning and the credential-automatic/key-customer-present split.
- CONTEXT: standing ruling S-33.
- STATUS: the headline change and the deletion STOP.
- REPORT-r204-item4.md rather than REPORT.md: a parallel session is active in this
  shared clone.
This commit is contained in:
2026-08-05 11:06:41 +02:00
parent 4114c5f891
commit 4faebe2926
7 changed files with 293 additions and 23 deletions
+46
View File
@@ -17,6 +17,52 @@
## Standing rulings ## Standing rulings
**S-33 — THE BOX DECLARES, THE HUB ANSWERS. R-204 item 4 / R-193's credential half closed
(2026-08-05, controller v0.199.0 + hub v0.96.0). Read with S-32; together they close all four of the
drill's manual interventions.**
**THE RULING AND ITS REASONING (operator, 2026-08-05).** Automate the rebuilt box's credential, and
**the trigger is a state the BOX DECLARES — never the hub inferring from a report.** From the hub an
ABSENT off-site object has FOUR meanings — never configured, mid-restart, a transient config read
failure, rebuilt-and-stranded — and the hub cannot tell them apart. The box can, from two local facts:
its data area is fresh (no repository password) AND the hub holds a sealed recovery package for it
(the ACK's `identity_blob_present`). **Both are required.** Freshness alone is a box that never had
off-site backups — declaring on it would make the whole fleet ask for credentials.
**SECOND RULING RECORDED:** the dashboard-password exposure on the future recovery-preview screen is
**metadata** (backup dates, app names), not content, and is **ACCEPTED**.
*Mechanism facts a future session needs:*
- **The ACK field used to be discarded.** `EscrowAutoConfirmer.Reconcile` returns early when the box is
neither pending nor escrowed — exactly a rebuilt box — so `identity_blob_present` was thrown away
every cycle. It is now recorded FIRST, before every gate, in the confirmer itself (already the one
place the ACK's escrow object arrives, and already wired; a second consumer would be a second wiring
point, and this project's built-but-never-wired count is six). Last-write-wins, not set-only: a
customer RESET must be able to turn the declaration back off.
- **The declaration is inert to every existing reader, established from their code:** it carries
`enabled:false` and zero sizes, and `OffsiteChecker.isStale` returns early on `!Enabled` while
`fillBand` returns OK on a zero quota/size. A configured box's report JSON is byte-identical to
v0.198.0's — no `state` key at all.
- **`reportHasOffsite` had to be tightened, and it is the ninth comment-vs-code entry — caught before
it shipped.** Its comment asserted *"presence == applied-on-the-box"*. Left as bare presence it would
have read a request for help as proof the tier was applied. It now requires `enabled:true`, provably
a no-op for every pre-v0.199.0 report shape.
- **Restage before mint is possible because the stored value SURVIVES a consume** —
`ConsumeOneTimeSecret` stamps `consumed_at` and nothing else, and `value` is `TEXT NOT NULL`. This was
established from the schema, NOT inherited from the PBS analogy; the two secrets are different objects
and assuming a shared shape is how two sessions confused the credentials.
- **The debounce is TWO DISTINCT REPORTS, derived from the ~15-minute report cadence** — one would act
on a blip, three would leave a stranded customer waiting ~45 minutes. It counts fresh evidence, not
reconciler ticks, so the 5-minute sweep cannot shorten it.
- **§8.4's deliberately-unhealed state exists and is excluded UPSTREAM, by construction:** the regressed
shape (a box that lost its target while still holding its repository password) cannot reach the
reconciler, because the declaration predicate requires the repository password to be absent.
- **R-192's counting guard is REPLACED, not repaired.** A declaration outranks both inferred shapes and
the delivery checker stands down with a record — two mechanisms healing one customer would
double-issue, and that one can only mint.
- **NEVER widen this to the ceremony. Credential automatic, key customer-present.**
**S-32 — THREE OF S-31's FOUR MANUAL STEPS ARE CLOSED (2026-08-05, R-204 items 13 / R-196). **S-32 — THREE OF S-31's FOUR MANUAL STEPS ARE CLOSED (2026-08-05, R-204 items 13 / R-196).
controller v0.198.0 + hub v0.95.0. Read this BEFORE S-31 — it supersedes S-31's steps 25.** controller v0.198.0 + hub v0.95.0. Read this BEFORE S-31 — it supersedes S-31's steps 25.**
+197
View File
@@ -0,0 +1,197 @@
# REPORT — R-204 item 4 / R-193 credential half (hub v0.96.0), 2026-08-05
**A rebuilt box asks for its credential back, and the hub answers.** The last of the four manual
interventions the 2026-08-04 drill needed. Controller half: `felhom-controller` v0.199.0.
> **Written as `REPORT-r204-item4.md`, not `REPORT.md`.** A PARALLEL SESSION is active in this shared
> clone — it committed `c917251` (the R-205…R-211 disk containment) between my baseline read and my
> first commit. Per `CLAUDE.md`'s parallel-session rule the second session never touches the shared
> `REPORT.md`. Explicit per-file staging was used throughout; verified after the fact that none of my
> three commits carries a foreign file.
## 1. Baselines, re-read on arrival — with a drift
| Repo | Expected | Found |
|---|---|---|
| `felhom.eu` | `0dbd954fec90` / hub v0.95.0 | **DRIFTED to `ee9d9bf`** — one docs-only commit ahead (R-205…R-211 spike output). Hub code, `manifests/hub.yaml` and the deployed image were all still v0.95.0, so the drift did not affect the work. A second foreign docs commit (`c917251`) landed mid-session. |
| `felhom-controller` | `68f195676b91` / v0.198.0 | exact match, tree clean |
| `felhom-agent` | v0.125.0 | `3f5f61b`, clean (rider only, **no version bump**) |
| `app-catalog-felhom.eu` | n/a | `122bbee`, clean (rider only, **no version bump**) |
**The task's "highest register ID in use: R-204" was STALE.** Grepped before minting, as instructed:
the highest is **R-211**. This session mints **R-212** (§8 below).
## 2. Part 2.0 FIRST — does the stored value survive a consume? **YES.**
Established from the schema and the code, deliberately **not** from the PBS analogy:
- `one_time_secrets` declares `value TEXT NOT NULL` (store.go, the CREATE TABLE).
- `ConsumeOneTimeSecret` runs `UPDATE one_time_secrets SET consumed_at = datetime('now')` — **it
touches nothing else.** The value column is never cleared or overwritten.
**So restage-before-mint is possible**, and `RestageOneTimeSecret` mirrors `RestageHostPBSSecret`:
clear the consumed flag, never touch the value, never bump a generation, return false when no row
exists so the caller escalates.
This is asserted rather than assumed by `TestRestageOneTimeSecret_ReArmsTheSameValue`, which checks
the **same** value comes back — so a future hardening that cleared the column fails loudly instead of
silently turning every rebuild into an external mint. The two secrets are different objects with
different lifecycles, and assuming a shared shape is how two earlier sessions confused the credentials.
## 3. The declared state — name, shape, and how a pre-upgrade hub treats it
**Name:** `needs_credential`, on the report's existing `offsite` object (not a new top-level field).
Constants: `backup.OffsiteStateNeedsCredential` / `offsiteheal.StateNeedsCredential`, pinned to each
other by `TestDeclaredStateStringMatchesTheReconciler`.
**Shape, as received by the hub live** (report id 16743):
```json
{"enabled": false, "escrow_state": "", "state": "needs_credential",
"snapshot_count": 0, "repo_size_bytes": 0, "quota_gb": 0}
```
**A pre-upgrade hub treats it as INERT, established from the readers' code rather than assumed:**
| Reader | Behaviour on the declaration | Why |
|---|---|---|
| `OffsiteChecker.isStale` | no staleness alarm | returns early on `!off.Enabled` |
| `OffsiteChecker.fillBand` | `bandOK` | returns OK on a zero quota/size |
| `encoding/json` | ignores `state` | unknown field, not an error |
| **`reportHasOffsite`** | **would have MISREAD it** | see below — the one reader that needed changing |
**A configured box's report JSON is byte-identical to v0.198.0's**`state` is `omitempty` and is
never set on a configured box (`TestOffsiteDeclare_ConfiguredBoxJSONIsUnchanged`).
## 4. The debounce: TWO distinct reports, derived not chosen
The controller reports every ~15 minutes. Two distinct declarations mean the state survived a full
report cycle, and a restart, a slow first report or a transient config read all resolve well inside
one. **One** would act on a blip; **three** would leave a genuinely stranded customer waiting ~45
minutes for the one thing they cannot obtain any other way. The reconciler's own sweep is 5 minutes —
deliberately faster than the report cadence so it adds no latency of its own — and the debounce counts
**fresh evidence, not ticks**, so a fast sweep cannot shorten it.
## 5. §8.4 — is there a deliberately-unhealed state here? **Yes, and it is excluded upstream.**
`pbsdrheal` refuses to heal `verify_failed` because re-staging would not help it and it must stay
loud. The off-site analogue is the **regressed** shape: a box that HAD a working tier and lost its
target while still holding its repository password. Re-arming a credential would not help it either.
**It cannot reach this reconciler at all** — the controller's declaration predicate requires the
repository password to be **absent**, so a box that still holds one never declares. The unhealable
case is excluded *by construction*, upstream, rather than filtered out in a switch. Stated explicitly
because "there is nothing like that here" is usually wrong, and this was checked rather than assumed.
## 6. Files modified
| File | Change |
|---|---|
| `hub/internal/store/store.go` | **new** `RestageOneTimeSecret`, **new** `LatestReportOffsiteDeclaration`; `reportHasOffsite` tightened to require `enabled:true` (+ its comment corrected) |
| `hub/internal/offsiteheal/reconciler.go` | **new package** — the reconciler |
| `hub/internal/offsiteheal/{reconciler,wiring}_test.go` | **new** — Scenarios AF + the wiring AST test |
| `hub/internal/store/offsite_restage_test.go` | **new** — the value-survives proof + the `reportHasOffsite` equivalence table |
| `hub/internal/monitor/offsite_delivery.go` | `shapeDeclared` outranks both inferred shapes; `maybeHeal` stands down with a record |
| `hub/internal/monitor/offsite_declared_test.go` | **new** — the demo-hp shape + the cross-package string pin |
| `hub/cmd/hub/main.go` | wires + runs the reconciler (`OFFSITEHEAL_ONLY_CUSTOMER` for a supervised rollout) |
| `.githooks/pre-push` | the workspace-root assertion (rider) |
| `manifests/hub.yaml` | image `0.95.0``0.96.0` |
**Commits on `main`:** `f62a115` (hub half) · `fe1e816` (rider) · `b2462b8` (CHANGELOG) ·
`4114c5f` (manifest). **Deployed:** `felhom-hub:0.96.0`, ArgoCD **Synced/Healthy**, `deploy/hub`
rolled out, startup log carries `offsite credential self-heal reconciler started (interval 5m,
debounce 2 reports)`.
## 7. Tests and red-proofs
Green gate: `go build ./... && go vet ./... && go test ./...` in `hub/`**rc=0**.
`python3 scripts/repo_gates.py --fast`**all five gates OK**.
| Scenario | Test | Result | Red-proof — what was mutated | Outcome |
|---|---|---|---|---|
| A+D | `TestScenarioAD_DeclaringBoxIsRestagedNotMinted` | PASS | inverted `heal()` so Reissue runs before Restage | **FAILED***"a sustained declaration was not served: restages=0"* |
| B | `TestScenarioB_SilentBoxIsNeverActedOn` | PASS | — | — |
| C | `TestScenarioC_HealthyBoxIsAPureNoOp` | PASS | — | — |
| E | `TestScenarioE_NothingToRestageEscalatesOnce` | PASS | — | — |
| F | `TestScenarioF_BlipIsAbsorbedByTheDebounce` | PASS | `debounceReportsDefault` 2 → 1 | **FAILED***"a one-report blip triggered a credential action"* (and Scenario A+D also failed, *"acted on a SINGLE declaration"*) |
| H | `TestMainWiresTheOffsiteHealReconciler` | PASS | (AST; comments dropped) | — |
| — | `TestRestageOneTimeSecret_ReArmsTheSameValue` | PASS | — | — |
| — | `TestReportHasOffsite_EnabledOnly` | PASS | — | — |
| — | `TestShapeOf_DeclarationOutranksBothInferredShapes` | PASS | — | — |
| — | `TestDisabledDescriptorIsNeverHealed`, `TestBlockedCustomerIsNeverHealed`, `TestRestageErrorDoesNotEscalate` | PASS | — | — |
**A test that first passed for the wrong reason, caught and fixed.** `TestBlockedCustomerIsNeverHealed`
initially seeded `Status: "blocked"` through `SaveCustomerConfig`, whose INSERT does not carry the
column — so the customer was never actually blocked and the assertion would have been vacuous. It now
goes through `SetCustomerConfigStatus` **and asserts `IsCustomerBlocked` before proceeding**.
## 8. Part 4 — HALTED AT ITS STOP. Nothing was deleted.
§8.7: *"If the paths do not match R-193's record exactly, STOP. A near-match on a protected endpoint
is not a match."* **They do not match.**
Measured read-only over SFTP, using each box's own credential, from inside its guest:
| Customer | Path | Size | What it is |
|---|---|---|---|
| demo-felhom (`u629488-sub1`) | `/home/felhom-repo` | **1.2 G** | **LIVE** — the configured `repo_path`. Unopenable by the box (R-193), but NOT a set-aside store |
| demo-felhom | `/home/felhom-repo.orphaned-20260717` | **1.4 G** | set aside |
| demo-felhom | `/home/felhom-repo.orphaned-20260718` | **3.0 M** | set aside |
| demo-hp (`u629488-sub3`) | `/home/felhom-repo` | **582 K** | **LIVE** |
| demo-hp | `/home/felhom-repo.orphaned-20260804` | **43 M** | set aside |
The ruling says *"~1.2 GB across the two demo boxes, in set-aside stores"*. Reality: **three** set-aside
stores totalling **~1.45 GB** — and **the figure that matches ~1.2 GB is demo-felhom's LIVE
`felhom-repo`**. Had the size been used to identify the target, the live repository would have been
deleted. **Filed as R-212, WAITING-ON-OPERATOR.**
## 9. Live validation
| # | What | Observable |
|---|---|---|
| 1 | **A healthy box: no action, no events** | demo-hp reported healthy throughout (`enabled:true`, no `state` key — report id 16742). Zero `offsite_selfheal_*` rows in the hub DB. **Honest limit:** the reconciler is silent by design on a healthy sweep, so there is no per-tick positive observable; what I have is the startup line proving `Run` was entered, the DB showing no events, and Scenario C. |
| 2 | **The declared state, produced live without wiping a box** | demo-felhom 9201 arranged **reversibly** into the stranded shape (settings + `offbox/` backed up first; the `offbox` key removed and `repo_password` moved aside). Report **id=16743** reached the hub carrying `{enabled:false, state:"needs_credential", quota_gb:0, repo_size_bytes:0}`. Restored the same minute; report **id=16744** is healthy again. **The single declaration was absorbed by the debounce — no self-heal event fired** — which is Scenario F demonstrated on live infrastructure rather than in a fake. |
| 2b | **The ACK field is no longer discarded** | Both demo boxes' `settings.json` now carry `hub_escrow_identity_present = true` — the recorder working on a HEALTHY box, which is the case that used to return early. |
| 3 | **A push from outside the workspace is refused** | A scratch clone at `/tmp/.../outside-clone`: `pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace`, naming `/mnt/5_hdd/felhom.eu`, **before the gates run**. **Red-proof:** with the assertion removed the same push **succeeded** (`rc=0`, new branch on a throwaway bare remote). In-workspace pushes ran normally all session. |
| 4 | **Part 4** | **Not performed — halted at the STOP.** Listings above. |
**Not fired live: the hub actually re-staging a credential.** Doing so would have re-applied
demo-felhom's off-site target mid-session and changed the very state Part 4's listing describes. It is
proven by Scenarios A/D/E with the store primitive proven separately against a real SQLite database.
**Teardown:** the scratch clone and throwaway remote are removed; demo-felhom's `settings.json` and
`offbox/` restored from the backup taken first (verified: `enabled=True`, `escrow_state=escrowed`,
`repo_password present=True`); the backup copy remains at `/root/r204-backup` on the guest for
traceability. Nothing else was provisioned.
## 10. Registers
- **R-204 — ALL FOUR ITEMS CLOSED.** Both 2026-08-05 rulings recorded on the row: the declared-state
trigger with its four-meanings-of-absence reasoning, and that the recovery preview's
dashboard-password exposure is **metadata, not content, and accepted**.
- **R-193 — credential half CLOSED.** Still open under this ID: the customer-facing recovery **screen**,
and the ciphertext deletion (now R-212).
- **R-192 — CLOSED, guard half by REPLACEMENT.** The counting inference is outranked by the declaration.
- **R-202 — untouched, still open.**
- **R-212 — NEW** (R-211 was the highest; grepped): the halted deletion, with the full measured listing.
## 11. CI
felhom-agent **0404f60**, app-catalog **ee2c810**, felhom-controller **992803c**, felhom.eu
**4114c5f** — run IDs and conclusions confirmed in the controller's `REPORT.md` §11 and re-checked at
session end. **`--no-verify` was NOT used**; every push ran the pre-push gate, including the new
workspace-root assertion.
## 12. Observations — noticed, NOT acted on
- **A second, accidental barrier exists outside the workspace and should not be relied on:**
`felhom-agent`'s `reuse-refs` gate FAILS in a clone outside the workspace because the shared
`reuse_refs_check.py` lives in the `felhom.eu` sibling. That is why the first Scenario-G red-proof
had to be redone with `app-catalog-felhom.eu`, whose fast gate is self-contained. It is incidental,
repo-specific and not a substitute for the assertion.
- **`offsite_credential_restaged` (the R-71c event) has still never fired for any customer.** The new
reconciler emits its own `offsite_selfheal_*` events, so the old one may now be permanently dead —
worth a deliberate look rather than leaving two event families for one concern.
- **demo-felhom's live `felhom-repo` is 1.2 GB of ciphertext its own box cannot open** (0 snapshots,
`last_status: error`). It is not a set-aside store, so it is out of R-212's scope as filed — but it is
the largest single block of unrecoverable data in the fleet and nothing currently plans its disposal.
+25 -14
View File
@@ -22,16 +22,16 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
## What's broken ## What's broken
- **A customer still cannot do that recovery alone — but only one step is left.** Getting from "the - **All four recovery steps are now automatic — but a customer still would not know to start.** The
key is recoverable" to "the file is back" took four steps that appeared in no instructions. **Three four steps that stood between "the key is recoverable" and "the file is back" are closed *(R-204)*:
are fixed today** *(R-204)*: the local reset-code tool works on the first try instead of needing the the reset-code tool works first time; re-issuing the storage credential no longer falsely marks the
controller restarted; re-issuing the storage credential no longer falsely marks the recovery key recovery key "stale"; the everyday restore says in plain Hungarian that it returned the app's
"stale" (which used to stop every off-site backup and invite the one act that would have destroyed settings and database and **not** your documents; and, as of today, **a rebuilt machine asks for its
the recovered key); and the everyday restore now **says in plain Hungarian that it returned the storage credential itself and the hub answers** — no Re-issue click. **What is missing is the
app's settings and database and not your documents**, and names the button that does. **The step offer:** there is no screen that meets the owner of a rebuilt machine, tells them a sealed package
that remains is the first one:** a rebuilt machine cannot get a storage credential by itself, is waiting, takes their recovery code and shows what would come back. So nothing needs *you* any
because the one-time password was used up by its predecessor — so you still have to press more, but it still needs someone who knows to look. **And the whole journey has not been re-run end
Re-issue. That is a design decision waiting on you, below. *(R-193)* to end since these fixes** — the four are proved one at a time, not as a single walk. *(R-193)*
- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key - **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one. Both demo that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one. Both demo
machines did this on 34 August — 51 backups (~1.2 GB) between them. The old key now survives the machines did this on 34 August — 51 backups (~1.2 GB) between them. The old key now survives the
@@ -47,6 +47,14 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
## What shipped recently ## What shipped recently
- **2026-08-05** — **A rebuilt machine now asks for its storage credential, and the hub gives it back.**
The machine says plainly what it needs — it can tell it has been rebuilt, because its data area is
empty *and* the hub is holding a sealed recovery package for it — instead of leaving the hub to guess
from a silence that has four possible meanings. The hub waits long enough to be sure it is not a
restart, then **re-uses the credential it already holds** before creating a new one at the storage
provider. **The recovery ceremony stays manual, deliberately:** a credential can be replaced, your
recovery code cannot. *(R-204 item 4, R-193)*
- **2026-08-05** — **The DooPlex server's disk is out of danger: 86% full → 54%, and the storage layer - **2026-08-05** — **The DooPlex server's disk is out of danger: 86% full → 54%, and the storage layer
is unstuck.** The cause was leftover working data from building our own software — 157 GB of it, is unstuck.** The cause was leftover working data from building our own software — 157 GB of it,
growing about 5 GB a day, which nothing was allowed to delete. **148 GB came back in 86 seconds.** growing about 5 GB a day, which nothing was allowed to delete. **148 GB came back in 86 seconds.**
@@ -99,10 +107,13 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
(controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing (controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing
was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against
199 GB now free, so this is about clutter, not space. *(R-210)* 199 GB now free, so this is about clutter, not space. *(R-210)*
- **The one-shot credential decision — this is now the last thing between a customer and an unaided - **The orphaned-backup deletion is STOPPED and needs your ruling — the list does not match.** You
recovery.** A rebuilt machine has no storage credential of its own, so an operator must press asked for about 1.2 GB in two set-aside stores to be deleted. Measured today, read-only: demo-felhom
Re-issue. Everything after that point is now self-service. Deciding how a rebuilt machine should get holds a **live** store of 1.2 GB plus set-aside stores of **1.4 GB** and **3 MB**; demo-hp holds a
a credential is the remaining design question. *(R-193, R-204 item 4)* live store of 582 KB plus a set-aside store of **43 MB**. So there are **three** set-aside stores
totalling ~1.45 GB, not two — **and the thing that is exactly 1.2 GB is demo-felhom's LIVE store**,
which must not be deleted. **Nothing was deleted.** Tell me which of the three `orphaned` stores to
remove. *(R-212)*
- **The recovery screen you described has been priced, and it can be built.** A freshly installed - **The recovery screen you described has been priced, and it can be built.** A freshly installed
machine that finds a sealed package waiting should say so, offer a box for the recovery code, and machine that finds a sealed package waiting should say so, offer a box for the recovery code, and
show what would come back before doing anything. One thing to weigh, deliberately not decided: that show what would come back before doing anything. One thing to weigh, deliberately not decided: that
File diff suppressed because one or more lines are too long
@@ -365,11 +365,26 @@ here — §8 has rows where it is the actual state.
| 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. | | 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. |
| 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed``OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. | | 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed``OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. |
| 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. | | 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. |
| 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** is required. | **OPEN → R-193.** This is a design decision (the one-shot credential model), not a defect, and it needs an operator ruling. **It is the reason the capability-map row keeps its "with a person present" qualifier.** | | 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** was required. | **CLOSED — controller v0.199.0 + hub v0.96.0** (2026-08-05, operator ruling: automate it, and **the trigger is a state the BOX DECLARES**). The box now reports `offsite.state=needs_credential` when two local facts hold together — a fresh data area AND a hub-held recovery package — and the hub's `internal/offsiteheal` re-arms the stored one-time secret, minting only when there is nothing to re-arm. **Deliberately NOT automated: the escrow ceremony.** A credential is replaceable; the recovery code is not. |
> **[FACT] The honest current answer:** three of the four are gone. **A customer alone still cannot > **[FACT] The honest current answer, updated 2026-08-05: all four are gone, and a customer alone
> complete a rebuild recovery**, because step 4 requires the operator. Everything after step 4 — > still does not have a guided recovery — for a different reason, which is worth keeping straight.**
> claim, restore, verify — is now self-service. > No step now REQUIRES an operator. What is missing is the OFFER: there is no customer-facing screen
> that tells a rebuilt box's owner a sealed package is waiting, takes their recovery code and previews
> what would come back (R-193's remaining half). The machinery is self-service; the experience is not
> yet built. **And the whole journey has not been re-walked end to end since these fixes** — the four
> closures are proven individually, not as one uninterrupted run.
>
> **WHY THE TRIGGER FOR STEP 4 IS A DECLARATION, recorded here because it is the design and not an
> implementation detail:** from the hub, an ABSENT off-site object means *never configured*,
> *mid-restart*, *a transient config read failure* OR *rebuilt and stranded*, and the hub cannot
> distinguish them. The BOX can, from two local facts it holds with certainty. So the box states its
> condition and the hub acts on a stated request — never on a silence. Both facts are required:
> freshness alone is a box that never had off-site backups, and an escrow alone is a healthy box.
>
> **The split that is deliberate and must not be widened: CREDENTIAL AUTOMATIC, KEY CUSTOMER-PRESENT.**
> A credential is transport and is replaceable; the recovery code is not, because only the customer
> holds it. Nothing in this chain runs, or asks for, an escrow ceremony.
### 7.1 The dependency graph ### 7.1 The dependency graph
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -66,7 +66,7 @@
| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN | | R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 | | R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
| R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 | | R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **ITEMS 13 SHIPPED (controller v0.198.0 + hub v0.95.0, 2026-08-05); ITEM 4 OPEN → R-193** | Collapsed per the coupling rule. The fix taken was R-196's — remove the guess, keep the two MEASURED signals (the controller's per-ACK hash re-check, which the stale flag was blinding, and R-197's supersession detector). Plus the claim-code read-through and the restore-scope wording. **Flips the capability-map recovery row from four crutches to one**, and that one is R-193's operator Re-issue, which keeps the row's *with a person present* qualifier. Does NOT touch R-198's retention, which is still unit-proven and needs the second drill | | R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **ALL FOUR ITEMS SHIPPED (controller v0.199.0 + hub v0.96.0, 2026-08-05)** | Collapsed per the coupling rule. The fix taken was R-196's — remove the guess, keep the two MEASURED signals (the controller's per-ACK hash re-check, which the stale flag was blinding, and R-197's supersession detector). Plus the claim-code read-through and the restore-scope wording. **Flips the capability-map recovery row from four crutches to one**, and that one is R-193's operator Re-issue, which keeps the row's *with a person present* qualifier. Does NOT touch R-198's retention, which is still unit-proven and needs the second drill **Item 4 shipped 2026-08-05 by DECLARATION, per the operator ruling: the box states its condition, the hub answers; an absence has four meanings and the hub cannot tell them apart. `internal/offsiteheal` mirrors `pbsdrheal` (declared states only, two-distinct-report debounce, restage before mint, healthy = no-op). Closes R-193's credential half and R-192's guard half (by replacement). **The capability-map row keeps a qualifier for a NEW reason** — no step needs an operator any more, but there is no customer-facing recovery SCREEN, so the experience still requires someone who knows to look. |
| R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet | | R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet |
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE | | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L11811219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L21462156, reinstall preserves existing), verified at L23322337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) | | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L11811219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L21462156, reinstall preserves existing), verified at L23322337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |