drill R-201: prepared and HALTED before the wipe — a mandatory data directory was missing from the off-site snapshot while the run said ok (R-203)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
This commit is contained in:
1 parent
846253baa8
commit
a0c4b607a6
7 files changed
+457
-190
No files matched your search
+21
@@ -17,6 +17,27 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-30 — the R-201 drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04). Nothing was wiped.**
|
||||
|
||||
It stopped at step 4 because the sentinel file was **not in the off-site snapshot** while the run
|
||||
reported `ok` (**R-203**): the deploy-time `${USERDATA_PATH}` root and the backup-time `NamespaceRoot`
|
||||
disagree for an app whose `HDD_PATH` equals `system_data_path` — and the same compose used both roots.
|
||||
Wiping would have destroyed the only copy of the sentinel and proven nothing.
|
||||
|
||||
*Established live on the way, all firsts, and all worth carrying:*
|
||||
- **A rebuilt box's off-site run REFUSES** — `repo_state: orphaned`, the orphan card, and
|
||||
`offbox_repo_orphaned` at the hub. It does **not** silently start a fresh history. Closes R-193's Q3.
|
||||
- **The orphan reset works**: move-aside to `/home/felhom-repo.orphaned-20260804`, never delete.
|
||||
- **demo-hp's pre-rebuild off-site history is permanently unrecoverable** — its key is in superseded
|
||||
row id 3 with `identity_blob` NULL, superseded **four hours before** v0.93.0 fixed the retention.
|
||||
- **Neither pre-existing off-site-toggled app has a restorable file leg** — both are named-volume-only,
|
||||
which the tier tars but the customer restore never unpacks. Any drill needing a recoverable file
|
||||
needs an app with a **mandatory userdata path**; `calibre-web` is now deployed on demo-hp as that
|
||||
fixture, with the sentinel in place.
|
||||
|
||||
*Still not established, unchanged:* **no file has ever been restored from an off-site backup after a
|
||||
wipe**, and Part 0's install path (controller v0.196.0) has never run against a live recovery.
|
||||
|
||||
**S-29 — a box may fetch its OWN sealed recovery blob with its OWN credential; the operator-driven DR
|
||||
path is a separate thing and stays gated (2026-08-04, R-199; hub v0.94.0 + agent v0.125.0 + controller
|
||||
v0.195.0).**
|
||||
|
||||
@@ -1,221 +1,168 @@
|
||||
# REPORT — the offsite backup key comes back (2026-08-04, R-199 closed, R-200 half)
|
||||
# REPORT — R-201 drill: PREPARED, HALTED BEFORE THE WIPE (2026-08-04)
|
||||
|
||||
**Class:** implementation across three repos + one decisive live proof. Nothing wiped, deleted,
|
||||
rotated or reconfigured; no customer-facing change in any language.
|
||||
**Nothing was wiped. Nothing irreversible happened.** The drill stopped at step 4 because the sentinel
|
||||
file was not in the off-site snapshot — wiping would have destroyed the only copy and proven nothing.
|
||||
|
||||
---
|
||||
|
||||
## 1. SCENARIO A — THE RESULT, FIRST, BECAUSE IT IS THE SESSION
|
||||
## 1. THE VERDICT — not reached, and that is the correct outcome
|
||||
|
||||
**On `demo-felhom`, 2026-08-04 13:49 CEST, through the real endpoints end to end:**
|
||||
The pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established the sentinel
|
||||
is not in the off-site snapshot at all**, so there was nothing to recover and the verdict is
|
||||
unobtainable. The runbook's own rule applies: *a precondition was wrong, and the finding outranks the
|
||||
drill.*
|
||||
|
||||
**Sentinel (step 3), still on the box:**
|
||||
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
|
||||
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt`, 181 B.
|
||||
|
||||
## 2. Snapshot count at step 9 — N/A; at step 4 it was 3, and the sentinel was in none of them
|
||||
|
||||
```
|
||||
=== offsite key recovery check (R-200) — compares, never installs ===
|
||||
on-disk sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
|
||||
recovered sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
|
||||
[MATCH] the offsite repository password IS recoverable from the sealed escrow.
|
||||
Nothing was written: this check compares and never installs.
|
||||
EXIT=0
|
||||
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
|
||||
snapshot_count 3 · repo_size_bytes 30 636
|
||||
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
|
||||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||
[INFO] [offbox] backed up calibre-web (…, 0 mandatory path(s))
|
||||
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
|
||||
```
|
||||
|
||||
**The hashes matched.** And they match a **third, independent** source: `host_escrow.restic_pw_sha256`
|
||||
on the hub reads `c60c8bc737a6…` for `demo-felhom-8363b5`, recorded at the ceremony and never involved
|
||||
in this comparison. The box's file, the sealed bundle and the hub's record all agree.
|
||||
**The run reported `ok`.** → **R-203**, §9.
|
||||
|
||||
**The offsite repository password is recoverable from the sealed escrow with the customer's recovery
|
||||
code.** That had never been demonstrated — the one live round-trip on record (2026-06-10) predates the
|
||||
field by a month.
|
||||
## 3. Preconditions P1–P6
|
||||
|
||||
## 2. Part 0 — the recovery code
|
||||
| # | Result |
|
||||
|---|---|
|
||||
| **P1** recovery code held | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file. The code appears nowhere in this report and **was never used** — no unseal ran on demo-hp. |
|
||||
| **P2** versions | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed published v0.125.0 (sha `f7d8339b53d9…`, verified against the release output) + controller v0.196.0. `age` present at `/usr/bin/age`. |
|
||||
| **P3** the blob seals the repo under test | **PASS with the caveat that reshaped the drill** — `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`; but **no repository existed under that key** (§5). |
|
||||
| **P4** verified rollback archive | **NOT TAKEN, deliberately** — only needed for the wipe, and the wipe did not happen. |
|
||||
| **P5** demo-felhom untouched | **PASS** — not touched at any point. |
|
||||
| **P6** space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 %, guest 64 GB free. |
|
||||
|
||||
**Answered by the operator before the session began:** the 2026-08-04 07:16/07:20 codes had not been
|
||||
kept, so **the operator ran a fresh ceremony on both nodes and saved the codes** (supplied to this
|
||||
session as `R_DEMO-FELHOM` in the DooPlex credentials file). No ceremony was run by CC.
|
||||
## 4. Every step's observable
|
||||
|
||||
Measured hub-side, and worth recording because it exercised two of yesterday's behaviours live:
|
||||
the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z` (demo-felhom), and
|
||||
`restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated them as
|
||||
same-password re-ceremonies: **no superseded row was created** and **no `offsite_repo_key_changed`
|
||||
fired**. That is v0.93.0's Scenario E, live. It also means yesterday's `identity_blob` retention is
|
||||
still unit-proven and structurally verified, **not yet exercised live** — no supersession has happened.
|
||||
| step | observable |
|
||||
|---|---|
|
||||
| 1 | `identity_blob` 572 B; `restic_pw_sha256` `8a9e33aa4da6…`; `snapshot_count: 0`, **no `last_run`/`last_status` at all** — never ran an off-site backup in this lifetime |
|
||||
| 1b | **repository ORPHANED** — see §5 |
|
||||
| 1c | **reset** (operator-authorised) — move-aside to `/home/felhom-repo.orphaned-20260804`, fresh repo initialised, `offbox_repo_reset` pushed |
|
||||
| 2 | recovery codes for both boxes created and saved by the operator earlier the same day; escrow rows stamped `11:11:37Z` / `11:13:06Z` with **`restic_pw_sha256` unchanged** on both — correctly treated as same-password re-ceremonies (**no superseded row, no `offsite_repo_key_changed`**): v0.93.0's Scenario E, live |
|
||||
| 3 | `calibre-web` deployed through the real API (HTTP 202), toggled off-site, Tier-1 recovery unit captured; sentinel written + `sync`ed + hashed |
|
||||
| 4 | `ok`, 3 snapshots, 30 636 B — **and the mandatory directory missing** |
|
||||
| 5–11 | **NOT RUN.** The §7 STOP was never reached: the drill failed its own precondition first |
|
||||
|
||||
Subject box: **demo-felhom** (Tier 0, reachable with a baked key, current escrow). Not demo-hp
|
||||
(break-glass only), not DooPlex, not ep0.
|
||||
## 5. Step 1b — the first live proof of a prediction, and it is the good news
|
||||
|
||||
## 3. The §8.2 decision as built, and where the gate is
|
||||
The 2026-08-04 spike predicted from source, and could not measure, that a rebuilt box's next off-site
|
||||
run would hit a **third** outcome — neither reattaching the old snapshots nor silently starting fresh,
|
||||
but **refusing**. Measured two ways:
|
||||
|
||||
**Built as §8.2:** `GET /api/v1/hosts/{host_id}/escrow`, authenticated with the box's own per-host key,
|
||||
self-scoped (global may read any — the same asymmetry the PUT has).
|
||||
**Read-only probe** (`restic cat config` with the current key, writes nothing):
|
||||
`Fatal: wrong password or no key found` — the exact string `classifyResticProbe` maps to `orphaned`.
|
||||
|
||||
**The single gate point is `api.escrowSelfServiceRetrieval`** (`hub/internal/api/handler.go`), a named
|
||||
constant. `false` re-imposes the recovery-mode requirement on this endpoint and changes nothing else —
|
||||
the same unseal, the same extraction, the same proof, minus the self-service property. The trade is
|
||||
written above the handler in `dr.go`'s style: what it gives out, why it is safe, and **what it changes
|
||||
about who is required** — obtaining the blob used to need an operator to arm recovery mode and now
|
||||
needs only that box's credential. They still cannot open it. The mitigation is the audit row.
|
||||
**Then the real customer path** (`POST /backup/offbox/run`):
|
||||
|
||||
## 4. Files, commits, deployed versions
|
||||
```
|
||||
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
|
||||
no-longer-available key; runs will skip until reset
|
||||
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
|
||||
[INFO] Event pushed: offbox_repo_orphaned (warning) …
|
||||
```
|
||||
|
||||
| Repo | Commits | Version | Deployed |
|
||||
|---|---|---|---|
|
||||
| `felhom.eu` | `435f4a5` (code) · `dd08926` (manifest) · docs commit below | hub **v0.94.0** | ArgoCD Synced/Healthy, `felhom-hub:0.94.0` |
|
||||
| `felhom-agent` | `6d79047` | **v0.125.0** | published sha `f7d8339b53d9…`, **verified by independent download**, installed on demo-felhom (`felhom-agent --version` → 0.125.0, `active`). **NOT vouched** |
|
||||
| `felhom-controller` | `9640e51` | **v0.195.0** | `felhom-controller:0.195.0` on guest 9201, `Up (healthy)` |
|
||||
`repo_state: orphaned`, orphan card rendered, event at the hub (HTTP 200).
|
||||
**The system stopped and said so. It did not quietly start a new history.** Closes R-193's Q3.
|
||||
|
||||
**Hub:** `internal/api/handler.go` (route + `handleHostEscrowGet` + `escrowSelfServiceRetrieval`),
|
||||
`internal/notify/dispatcher.go` (`escrow_blob_served` → `operatorOnlyEvents`),
|
||||
`internal/api/escrow_get_test.go` (new).
|
||||
**Agent:** `internal/escrow/recover.go` (new — `OffsiteKeyRecoverer`, distinct errors),
|
||||
`internal/hub/client.go` (`FetchIdentityEscrow`), `internal/localapi/escrow_recover.go` (new route),
|
||||
`internal/localapi/server.go` (seam + registration), `cmd/felhom-agent/main.go` (production wiring +
|
||||
the §8.6 selftest message fix), plus `recover_test.go` and `escrow_recover_wiring_test.go` (new).
|
||||
**Controller:** `internal/backup/offbox_recovery_check.go` + `offbox_recovery_cli.go` (new),
|
||||
`internal/agentapi/escrow.go` (`RecoverOffsiteRepoPassword`), `cmd/controller/main.go`
|
||||
(`--recover-offsite-check`), plus `offbox_recovery_check_test.go` (new).
|
||||
**Why it was orphaned:** the 15 snapshots / 40.9 MB were written under key `8e03eddf9ff7…` before the
|
||||
2026-08-03 rebuild. That key survives only in superseded escrow row **id 3**, whose `identity_blob` is
|
||||
**NULL** — superseded at `2026-08-04 07:15:36`, **four hours before hub v0.93.0 fixed the retention**.
|
||||
Permanently unrecoverable, with or without a recovery code.
|
||||
|
||||
**Green gate in each repo after each phase** (`go build && go vet && go test ./...`, rc=0), plus
|
||||
`repo_gates.py` / `agent_gates.py` / `controller_gates.py` — all OK. **No test run was combined with a
|
||||
commit.**
|
||||
## 6. Step 7's assertion (`identity_blob` unchanged across the rebuild) — NOT REACHED
|
||||
|
||||
## 5. Tests and every red-proof
|
||||
There was no rebuild. Unmeasured, and stated as such.
|
||||
|
||||
| Scenario | Result | Red-proof — mutation → outcome |
|
||||
## 7. Did the built chain carry steps 8–9? — NOT REACHED
|
||||
|
||||
Neither the check nor the install ran against a live recovery on demo-hp. The manual substitute was not
|
||||
needed and was not used. **Part 0 remains unit-proven only.**
|
||||
|
||||
## 8. Part 0 — files, commit, tests, deployment
|
||||
|
||||
**Commit `1b1366b`**, controller **v0.196.0**, deployed to demo-hp guest 9201 (`Up (healthy)`).
|
||||
`internal/backup/offbox_recovery_cli.go` (`RecoverAndInstall`), `cmd/controller/main.go`
|
||||
(`--recover-offsite-install` + `--confirm-install`), `internal/backup/offbox_recovery_check_test.go`.
|
||||
|
||||
`go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK. Test run and commit were
|
||||
never combined.
|
||||
|
||||
**Red-proof observed:** removing the confirmation gate makes the dry run write the password —
|
||||
`TestRecoverAndInstall_InstallsOnABareBox` fails with *"the DRY RUN wrote the password"*.
|
||||
|
||||
## 9. R-203 — the finding that halted the drill
|
||||
|
||||
| what | path | exists? |
|
||||
|---|---|---|
|
||||
| **A** recovered key == on-disk key | PASS (unit, real `age`) + **PROVEN LIVE** | return `bundle.PBSToken` instead of `ResticRepoPassword` → **FAIL**: *"the recovered key is not the sealed repository password (len 24 vs 64)"*. This is the mutation that mattered: every field of that bundle is a non-empty string that looks like a secret |
|
||||
| **B** wrong R fails closed, nothing written | PASS + **PROVEN LIVE** | — (the property is age's KDF, asserted rather than guarded, so there is no guard to remove) |
|
||||
| **C** a box fetches only its own blob | PASS | delete the `!isGlobal && authHostID != pathHostID` check → **FAIL**: host A was served host B's blob, body and all |
|
||||
| **D** no blob → a clean "none" | PASS | — |
|
||||
| **E** every retrieval leaves a record | PASS + **PROVEN LIVE** | `if false &&` on the audit block → **FAIL**: *"a sealed recovery blob was served and NOTHING recorded it"* |
|
||||
| **F** R persists nowhere | PASS + **PROVEN LIVE** | write R to a temp file inside the unseal → **FAIL** (see the correction below) |
|
||||
| **G** the operator DR path is untouched | PASS | — |
|
||||
| **H** the seam is wired (AST) | PASS | `_ = escrowRecoverer` with the `Options` field removed → **FAIL**: *"localapi.Options … carries no EscrowRecovery field"* |
|
||||
| §8.5 compare-never-install | PASS + **PROVEN LIVE** | add `InjectOffboxPassword` to the check → **FAIL**: *"the check INSTALLED the recovered password"* |
|
||||
| the app's live bind (where the sentinel is) | `/mnt/sys_drive/userdata/media/books` | **YES** |
|
||||
| what the off-site capture set looked for | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
|
||||
|
||||
**TWO MUTATIONS DID NOT REMOVE EVERY GUARD ON THE FIRST ATTEMPT, AND BOTH ARE RECORDED RATHER THAN
|
||||
QUIETLY REDONE** (§9.13 — two of the last four sessions found a red-proof that passed):
|
||||
**Mechanism, from source:** `NamespaceRoot` (`appbackup/paths.go:28-33`) appends `felhom-data` **when
|
||||
the drive IS the system data path** (`backup/backup.go:331`); the deploy-time `${USERDATA_PATH}` is
|
||||
`<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an
|
||||
app at `HDD_PATH=/mnt/sys_drive`, they differ. **The same compose used both roots** —
|
||||
`${IMPORT_PATH}` → `…/felhom-data/userdata/import/calibre`, `${USERDATA_PATH}` →
|
||||
`/mnt/sys_drive/userdata/media/books`.
|
||||
|
||||
1. **Scenario F.** The planted leak wrote R into the walked directory and **the test still passed** —
|
||||
because a later failing call OVERWROTE the leak file with the wrong code, and the test only scanned
|
||||
for the first one. Instrumenting the mutation (printing where it wrote) is what found it. **The test
|
||||
was wrong, not the mutation.** It now asserts **emptiness** of the TMPDIR tree as its primary check —
|
||||
nothing under it is created by the test, so any survivor is a leak regardless of content — with the
|
||||
content scan kept as defence in depth over all three secrets. Re-run: **FAIL**, *"the unseal left
|
||||
1 file(s) behind under TMPDIR"*.
|
||||
2. **Scenario H.** Commenting the wiring line out failed to **compile** (`declared and not used`), which
|
||||
is not a red-proof. The mutation that compiles keeps the variable alive (`_ = escrowRecoverer`) and
|
||||
drops the struct field. Re-run: **FAIL** as quoted above.
|
||||
**Measured vs not, because it changes the fix.** MEASURED: the disagreement, the absence from the
|
||||
snapshot, the `ok` status, and that the only signal is a container-log WARN. **NOT ESTABLISHED:**
|
||||
whether `HDD_PATH=/mnt/sys_drive` is supported — it was chosen because demo-hp's only registered drive
|
||||
(`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working), while
|
||||
`/mnt/sys_drive` was **accepted** (202). **Either branch is a defect** and the row says so.
|
||||
|
||||
**Every `-run` filter was verified to have matched** (`-v`, `=== RUN` lines counted) before any result
|
||||
was read as a proof.
|
||||
**Not a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths, so
|
||||
they are unaffected and their snapshots are real.
|
||||
|
||||
## 6. The live validation trail, in order
|
||||
## 10. Teardown — three layers, nothing torn down that proves anything
|
||||
|
||||
1. **Wrong code first, deliberately** (13:44:44) — hub: `escrow blob SERVED to host demo-felhom-8363b5
|
||||
(572 opaque bytes, self_scope=true) — recovery path in use`; agent: `offsite key recovery FAILED …
|
||||
err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
|
||||
blob): exit status 1"`; CLI: `[FAIL] … nothing was written.`, exit 1.
|
||||
**This is Scenario B live AND it proves links 6 and 7 ran independently of the success** — the blob
|
||||
was fetched and the unseal was attempted and refused.
|
||||
2. **The real code** (13:49:47) — §1's output, exit 0.
|
||||
3. **Nothing was written** — `offbox/repo_password` mtime on the live box is still
|
||||
`2026-08-03 07:18:02`, unchanged by a check that ran at `11:49` UTC. The whole `offbox/` directory is
|
||||
byte-for-byte the pre-check listing.
|
||||
| layer | state |
|
||||
|---|---|
|
||||
| the guest | `calibre-web` deployed, running, off-site-toggled, sentinel in place. **Kept** — it is the only app on either demo box with a restorable file leg, and the resumed drill needs it |
|
||||
| the off-site store | fresh, working, 3 snapshots, 30 636 B. Old 15-snapshot history **set aside, not deleted**, at `/home/felhom-repo.orphaned-20260804` (≈41 MB) |
|
||||
| the host | agent v0.125.0, controller v0.196.0; `pvesm` unchanged beyond normal usage |
|
||||
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` recorded for `demo-hp`. **No scratch customer records were created — nothing was reinstalled** |
|
||||
|
||||
## 7. The retrieval record as it appears at the hub
|
||||
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — ruled, still owed, and §8.3 forbids
|
||||
riding it along with a drill.
|
||||
|
||||
```
|
||||
customer_id severity created_at details_json
|
||||
demo-felhom warning 2026-08-04 11:44:44 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
|
||||
demo-felhom warning 2026-08-04 11:49:47 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
|
||||
```
|
||||
## 11. The capability-map row as it now reads
|
||||
|
||||
And the routing, which is the half that shows the register works:
|
||||
The R-199 row (the KEY is recoverable, PROVEN-LIVE) keeps its scope and gains the back-pointer the
|
||||
previous update omitted, plus: the attempt to prove the DATA half was **halted before the wipe**, and
|
||||
**no file has still ever been restored from an off-site backup after a wipe**. The install half
|
||||
(v0.196.0) is likewise unit-proven only.
|
||||
|
||||
```
|
||||
customer_id channel status created_at
|
||||
demo-felhom operator sent 2026-08-04 11:44:44
|
||||
demo-felhom customer skipped 2026-08-04 11:44:44 ← operator_only
|
||||
demo-felhom operator suppressed 2026-08-04 11:49:47 ← 1h operator cooldown, itself recorded (R-182)
|
||||
demo-felhom customer skipped 2026-08-04 11:49:47
|
||||
```
|
||||
## 12. Registers
|
||||
|
||||
**Both retrievals raised an event; the operator was mailed for the first; the second's mail was
|
||||
cooldown-suppressed and that suppression is written down.** The audit is per-retrieval and complete
|
||||
even where the mail is not.
|
||||
- **R-203 NEW** — the missing mandatory directory. Highest ID before this session was R-202.
|
||||
- **R-201** — prepared, halted, with everything it established and exactly what resuming needs.
|
||||
- **R-200** — plumbing complete (v0.196.0); the form is not built; the install has never run live.
|
||||
- **R-202** stays open. **The ciphertext deletion is still owed.**
|
||||
- **The v0.93.0 `identity_blob` retention is still unit-proven only** — nothing here superseded a key.
|
||||
|
||||
## 8. Evidence that R persisted nowhere — the search, not a claim
|
||||
## 13. CI
|
||||
|
||||
Searched on the subject host with the pattern read from a file (never in argv), across the agent
|
||||
journal, the controller's container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`:
|
||||
Run numbers and task ids in the session summary; **`--no-verify` was not used**.
|
||||
|
||||
```
|
||||
agent journal lines containing R: 0
|
||||
controller log lines containing R: 0
|
||||
host files containing R: 0
|
||||
leftover felhom-idesc-* staging dirs: 0
|
||||
staged escrow secret file: 0 entries
|
||||
```
|
||||
## 14. Observations — noticed, NOT acted on
|
||||
|
||||
**With a positive control, because an absent match is not a measurement unless the instrument is shown
|
||||
to work:** a copy of R was planted under `/tmp`, the same sweep found **1**, the copy was shredded, and
|
||||
the sweep returned **0**. The instrument detects R when R is there.
|
||||
|
||||
R reached the box only as: the credentials-file value on DooPlex → a pipe → the container's stdin →
|
||||
the request body → the agent's memory. It was never an argument, never a file on either machine, and
|
||||
appears in no log line at any level.
|
||||
|
||||
## 9. Register
|
||||
|
||||
- **R-199 → SHIPPED + PROVEN-LIVE.** Links 6, 7 and 8 assembled and walked.
|
||||
- **R-200 → plumbing shipped; the FORM is not built** and the row stays open for it and for link 9.
|
||||
- **R-201 → OPEN, and the distinction is deliberate:** the KEY is proven recoverable; a recovered
|
||||
password has never been **installed**, no repository has been **reopened** under one, and **no file
|
||||
has ever been restored**. The drill's pass condition is unchanged — a byte-identical sentinel file,
|
||||
not "the store opened".
|
||||
- **R-202 → OPEN and untouched.** The orphan card still promises recoverability unconditionally.
|
||||
- **The orphaned-ciphertext deletion (~1.2 GB) is STILL OWED** — operator ruling 2 of 2026-08-04,
|
||||
deliberately not ridden along with a code release.
|
||||
- Capability map, `03-host-agent.md` §8a, `CONTEXT.md` **S-29**, `REUSE.md` and `ROADMAP.md` updated;
|
||||
`STATUS.md` rewritten for the operator.
|
||||
|
||||
## 10. The capability-map row, and what it deliberately does not claim
|
||||
|
||||
Added: *"The offsite repository password can be RECOVERED from the sealed escrow with the customer's
|
||||
recovery code"* — **PROVEN-LIVE (2026-08-04)**, with the evidence above.
|
||||
|
||||
**What the row explicitly refuses to claim**, stated in the row itself because the previous over-claim
|
||||
on this subject was struck out four hours earlier: it covers the **key**, not the **data**. No
|
||||
recovered password has been installed, no repository reopened, no file restored. And the proof used a
|
||||
box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is
|
||||
exactly what the drill covers and it has not run.
|
||||
|
||||
## 11. CI
|
||||
|
||||
Run numbers, task ids and conclusions for all commits are quoted in the session summary.
|
||||
**`--no-verify` was NOT used** — every push ran its repo's pre-push gate and passed.
|
||||
|
||||
## 12. Teardown
|
||||
|
||||
**Nothing was provisioned.** The temporary pattern file used by the R sweep and its positive control
|
||||
were shredded on the host by the same command that created them (verified: 0 files remain). No
|
||||
diagnostic artifact persists on either machine. The agent's previous binary is retained as
|
||||
`/usr/local/bin/felhom-agent.bak-0.124.1`, per the documented deploy.
|
||||
|
||||
## 13. Observations — noticed, NOT acted on
|
||||
|
||||
1. **The `escrow_blob_served` mail is subject to the 1-hour operator cooldown** — visible above, where
|
||||
the second retrieval's mail was suppressed. The EVENT rows are per-retrieval so the audit is
|
||||
complete, but if retrieval ever becomes routine, a burst would produce one mail. Worth revisiting
|
||||
**when** the customer-facing flow lands, not before.
|
||||
2. **`ListSupersededEscrow` still has no production caller.** Recovery reads the CURRENT blob; a
|
||||
customer needing a PREVIOUS generation's key (which is what R-198's retention exists for) has no
|
||||
path at all. That is the next gap in this area after the drill.
|
||||
3. **The controller binary is at `/usr/local/bin/felhom-controller`, not `/app/…`** — the first
|
||||
diagnostic invocation failed on a guessed path. Worth a line in the runbook when one is written.
|
||||
4. **`age` is at the hardcoded `/usr/bin/age` on both demo hosts** (1.2.1 on demo-felhom) — the recon
|
||||
listed this as unestablished; it is now established for demo-felhom. `ageBinary` is a package
|
||||
variable with no config override, so a host that installs `age` elsewhere would fail the unseal at
|
||||
the last step.
|
||||
5. **Yesterday's `identity_blob` retention has still not been exercised live** — the operator's fresh
|
||||
ceremonies sealed the *same* password, so no supersession occurred. It remains unit-proven with a
|
||||
verified schema; the first real supersession will be its live proof.
|
||||
1. **`${IMPORT_PATH}` and `${USERDATA_PATH}` resolved to different roots in one deploy** — the sharpest
|
||||
single expression of R-203, and the place a fix should probably start.
|
||||
2. **The controller's CSRF form field is `_csrf`, not `csrf_token`** — the login page carries one name
|
||||
and the protected forms another; an hour went into that. Worth a line in the headless-access memory.
|
||||
3. **`/api/stacks/<name>/deploy` accepted `HDD_PATH=/mnt/sys_drive`** one call after refusing the NAS
|
||||
path with a clear Hungarian error. If the system drive is not a valid app namespace, that refusal is
|
||||
missing.
|
||||
4. **demo-hp has no registered drive usable as an app namespace** — its only drive is a NAS. Any drill
|
||||
needing a file-leg app on that box is constrained by this.
|
||||
5. **The `felhom-r-leak`-style positive control is now used in three separate absence checks** across
|
||||
two sessions. It is cheap and it has already caught one wrong test; worth making a documented habit.
|
||||
@@ -46,6 +46,16 @@ Proven end to end on real hardware.
|
||||
recovered key back*, reopens the old backup store with it, or **restores a single file**. Today
|
||||
proves the key survives and returns; it does not prove the backups do. *(R-199 closed; R-200 half;
|
||||
R-201 open)*
|
||||
- **A backup reported success while leaving out a folder the customer was told is protected.** Found
|
||||
this evening on the HP machine, while setting up the wipe-and-restore proof. We deployed an app whose
|
||||
book folder is declared *mandatory* — the strongest protection class — put a marked file in it, and
|
||||
ran an off-site backup. **The backup said OK. Three snapshots. The folder was not in any of them.**
|
||||
The machine knew: one warning line inside the container says the folder was skipped. Nothing else
|
||||
does — not the card, not the counters, not the hub, not you. This is the same shape as everything
|
||||
else we have been fixing this month: *a path the customer thinks is protected is not in the copy.*
|
||||
It is the reason the proof stopped before the wipe — wiping would have destroyed the marked file and
|
||||
proven nothing. **The two apps that were already backing up off-site are unaffected** — they declare
|
||||
no such folders. *(R-203)*
|
||||
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
|
||||
says the old backups may later be restorable with the matching recovery code. From today that is true
|
||||
for machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||||
@@ -94,10 +104,9 @@ Proven end to end on real hardware.
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** the proof exercise — wipe a demo machine and recover it with a saved recovery code. It is
|
||||
designed, and it is now a much better bet than it was this morning: the first half of the path was
|
||||
walked live today, so if the drill fails we will know *which* step failed instead of just "recovery
|
||||
did not work". Waiting on your go-ahead. Both honesty fixes shipped today: a changed backup key now
|
||||
- **Now:** fixing the folder-left-out-of-the-backup problem above. The wipe-and-restore proof is
|
||||
**staged and waiting on it** — the machine, the code, the working off-site store, the app and the
|
||||
marked file are all in place; only the missing folder blocks it. Nothing was wiped. Both honesty fixes shipped today: a changed backup key now
|
||||
raises an alarm on the day, and the email that stated the opposite of what it measured now describes
|
||||
what it actually saw.
|
||||
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
|
||||
@@ -149,6 +158,12 @@ Proven end to end on real hardware.
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-04 (late)** — Set up the wipe-and-restore proof on the HP machine and **stopped before the
|
||||
wipe**: a folder marked as protected was missing from the off-site backup while the backup reported
|
||||
success. Three things were proved on the way, all firsts: a rebuilt machine's off-site backup now
|
||||
**refuses and says so** instead of quietly starting over; the "start a new store" repair works and
|
||||
keeps the old data aside; and the HP machine's pre-3-August history is gone for good — its key was
|
||||
destroyed four hours before the fix that would have kept it. *(R-203, R-201, R-193)*
|
||||
- **2026-08-04 (evening)** — **Proved the backup key comes back.** A demo machine fetched its own
|
||||
sealed package, opened it with the saved recovery code, and produced a key identical to the one it
|
||||
uses. A wrong code was refused and wrote nothing. The recovery code left no trace anywhere on the
|
||||
|
||||
@@ -38,7 +38,7 @@
|
||||
| Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention) — ⚠ the claim as written was FALSE for the offsite tier for two months; **FIXED in hub v0.93.0 (2026-08-04), and the row below states what ships TODAY:** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`) — and **since v0.93.0 the retained row carries `identity_blob` as well as the K-escrow blob**, so a previous passphrase does now stay recoverable with the recovery code that sealed it. **What was wrong until v0.93.0, recorded because it is the ninth entry in `CLAUDE.md`'s comment-vs-code table and the first that was also customer-facing copy:** `host_escrow_superseded` had no `identity_blob` column and `demoteCurrentEscrowTx` did not copy one, so what survived a supersession was the **PBS datastore key only** — never the restic repository password, which lives in `identity_blob`. The destroying act was the **escrow ceremony** a rebuilt box asks its customer to run. Measured live 2026-08-04, before the fix: both current rows held `blob`=383 B **and** `identity_blob`=572 B; both retained rows held `blob`=383 B only. → **R-198** (SHIPPED), evidence `audits/RECON-offsite-dr-chain-2026-08-04.md` §7. **THREE SCOPE LIMITS THIS ROW MUST NOT BE READ PAST.** (1) **Nothing was backfilled and nothing could be** — rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; both demo boxes' pre-2026-08-04 repository passwords are gone permanently. (2) **A retained key is not a restore — but as of 2026-08-04 evening it IS a recovered key.** See the row below. (3) The customer-facing orphan card still promises recoverability unconditionally (R-202, gate hit, card untouched). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
|
||||
|
||||
| **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the Line truncated
|
||||
| **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. **Re-confirmed 2026-08-04 evening by an attempt to prove the DATA half:** the R-201 drill was prepared on demo-hp and **halted before the wipe** — the sentinel file was not in the off-site snapshot (R-203), so the wipe would have destroyed it and proven nothing. **No file has still ever been restored from an off-site backup after a wipe** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). The install half of the chain (controller v0.196.0 `--recover-offsite-install`, R-200) is likewise unit-proven only — it has never run against a live recovery. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction);Line truncated
|
||||
| **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** |
|
||||
| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **THE HOST TIER IS NOW PROVEN-LIVE, UNATTENDED, ON BOTH DEMO BOXES (2026-08-0Line truncated
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below |
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
# DRILL RECORD — R-201, the wipe-and-recover proof: **PREPARED, HALTED BEFORE THE WIPE**
|
||||
|
||||
**Date:** 2026-08-04 · **Box:** `demo-hp` (Tier 0, the designated drill host) · **Nothing was wiped.**
|
||||
**Outcome:** the drill did not reach its verdict. It was halted at step 4 by a defect that makes the
|
||||
verdict unobtainable — and that defect is worth more than the drill.
|
||||
|
||||
> **THE HEADLINE.** A customer-declared **mandatory** data directory was **silently absent from the
|
||||
> off-site snapshot**, while the backup reported `ok` with three snapshots. The only trace is one
|
||||
> `[WARN]` line inside the controller container. The hub, the card and the counters all say the backup
|
||||
> succeeded. → **R-203**
|
||||
>
|
||||
> **Nothing irreversible happened.** The wipe never ran. `demo-hp` is left in a BETTER state than it
|
||||
> started: it now has a working off-site repository (it had an orphaned one), three snapshots, and a
|
||||
> sentinel file on disk.
|
||||
|
||||
---
|
||||
|
||||
## 1. The verdict — not reached, and why that is the correct outcome
|
||||
|
||||
The drill's pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established that
|
||||
the sentinel is not in the off-site snapshot at all.** Wiping the box would therefore have:
|
||||
|
||||
- destroyed the sentinel, which exists only on that box;
|
||||
- proven nothing about recovery, because there would be nothing to recover;
|
||||
- and done so *after* the point of no return.
|
||||
|
||||
The runbook's own rule applies: *"If this session finds itself writing code beyond Part 0, stop. That
|
||||
means a precondition was wrong, and the finding outranks the drill."* A precondition was wrong. It was
|
||||
one the runbook's P1–P6 table did not contain, because nobody knew to look for it.
|
||||
|
||||
**Sentinel sha256 (step 3), recorded and still on the box:**
|
||||
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
|
||||
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt` (181 B).
|
||||
|
||||
---
|
||||
|
||||
## 2. R-203 — the defect that halted the drill
|
||||
|
||||
**Measured, twice, on the live box.**
|
||||
|
||||
| what | path | exists? |
|
||||
|---|---|---|
|
||||
| the app's live bind, where the customer's files actually land | `/mnt/sys_drive/userdata/media/books` | **YES** (the sentinel is here) |
|
||||
| the path the off-site capture set treats as the mandatory directory | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
|
||||
|
||||
The controller's own log, verbatim:
|
||||
|
||||
```
|
||||
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
|
||||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||
[INFO] [offbox] backed up calibre-web (…/backups/primary/calibre-web, 0 mandatory path(s))
|
||||
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
|
||||
```
|
||||
|
||||
**The run reported `ok`.** `last_status: ok`, `last_success` stamped, `snapshot_count: 3`. Nothing
|
||||
customer-visible, nothing hub-visible and nothing in any counter says the mandatory directory was
|
||||
dropped. This is `CLAUDE.md`'s recurring shape — *a path the customer thinks is protected is not in the
|
||||
snapshot* — and the code even has the right words for it in a WARN nobody reads.
|
||||
|
||||
**The mechanism, from source.** `NamespaceRoot(drivePath, inGuestDrive)`
|
||||
(`appbackup/paths.go:28-33`) appends `felhom-data` **when the drive IS the system data path**
|
||||
(`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`,
|
||||
`backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is documented and computed
|
||||
as **`<HDD_PATH>/userdata`** (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive`
|
||||
and an app deployed at `HDD_PATH=/mnt/sys_drive`, those two produce different directories.
|
||||
|
||||
**And the same compose file used BOTH roots.** From `docker inspect calibre-web`:
|
||||
|
||||
```
|
||||
bind /mnt/sys_drive/felhom-data/userdata/import/calibre -> /cwa-book-ingest ← felhom-data root
|
||||
bind /mnt/sys_drive/userdata/media/books -> /calibre-library ← NO felhom-data root
|
||||
```
|
||||
|
||||
`${IMPORT_PATH}` resolved *with* the segment; `${USERDATA_PATH}` resolved *without* it. One deploy, one
|
||||
template, two roots.
|
||||
|
||||
### What is measured and what is not — stated because the scope changes the fix
|
||||
|
||||
- **MEASURED:** the two paths disagree; the mandatory directory is absent from the snapshot; the run
|
||||
reports `ok`; the only signal is a container-log WARN.
|
||||
- **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a *supported* choice. It was chosen because
|
||||
demo-hp's only registered drive, `Felhom-Share`, is a NAS and was **correctly refused** as an app
|
||||
namespace (R-108's `RefuseAsAppNamespace`, working as designed). `/mnt/sys_drive` was **accepted**
|
||||
(HTTP 202) rather than refused.
|
||||
|
||||
**Either branch is a defect, which is why this is filed regardless:**
|
||||
|
||||
- if the system drive **is** a supported app namespace → the userdata path resolution is wrong for
|
||||
every app deployed on it, and their mandatory directories are silently unprotected;
|
||||
- if it is **not** supported → the deploy accepted a namespace it should have refused, exactly as it
|
||||
refused the NAS one call earlier, and the refusal that exists is not reaching this case.
|
||||
|
||||
**What must NOT be concluded from this drill:** that off-site backups are broken generally. The two
|
||||
pre-existing apps (`opengist`, `privatebin`) declare **no** mandatory userdata paths — everything they
|
||||
own is in named volumes — so they are unaffected, and their snapshots are real.
|
||||
|
||||
---
|
||||
|
||||
## 3. Preconditions, each measured
|
||||
|
||||
| # | Precondition | Result |
|
||||
|---|---|---|
|
||||
| **P1** | operator holds the recovery code | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file (`~/.config/credentials`). *The code itself appears nowhere in this record.* |
|
||||
| **P2** | demo-hp on agent v0.125.0 + controller v0.196.0 | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed the published v0.125.0 (sha `f7d8339b53d9…`, verified against the release) and controller v0.196.0. `age` present at `/usr/bin/age`. |
|
||||
| **P3** | the current `identity_blob` seals the repository under test | **PASS with a caveat that reshaped the drill** — `identity_blob` = 572 B, `restic_pw_sha256` = `8a9e33aa4da6…`. But **no repository existed under that key** (see §4). |
|
||||
| **P4** | a deliberate, verified whole-guest archive as rollback | **NOT TAKEN** — deliberately. It is only needed for the wipe, and the wipe did not happen. |
|
||||
| **P5** | demo-felhom untouched and healthy | **PASS** — not touched at any point. |
|
||||
| **P6** | space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 % used, guest 64 GB free. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Step-by-step, with every observable
|
||||
|
||||
### Step 1 — starting state (hub, read-only)
|
||||
|
||||
`demo-hp-bb76ea`: `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, created `2026-08-04T11:11:37Z`, `escrow_state: escrowed`.
|
||||
Report: `snapshot_count: 0`, `repo_size_bytes: 0`, **no `last_run`/`last_status` at all** — the shape of
|
||||
a controller that has never run an off-site backup in this lifetime.
|
||||
|
||||
### Step 1b — the repository was ORPHANED, and this is the first live proof of a prediction
|
||||
|
||||
The 2026-08-04 spike predicted from source — and could not measure — that a rebuilt box's next
|
||||
off-site run would hit a **third** outcome: neither reattaching the old snapshots nor silently starting
|
||||
a fresh history, but **refusing**. Measured here, twice over.
|
||||
|
||||
**Read-only probe first** (`restic cat config` with the current key, writes nothing):
|
||||
|
||||
```
|
||||
Fatal: wrong password or no key found
|
||||
```
|
||||
|
||||
— the exact string `classifyResticProbe` maps to `"orphaned"`.
|
||||
|
||||
**Then the real run, through the customer UI endpoint** (`POST /backup/offbox/run`):
|
||||
|
||||
```
|
||||
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
|
||||
no-longer-available key; runs will skip until reset
|
||||
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
|
||||
[INFO] Event pushed: offbox_repo_orphaned (warning) — A távoli mentési tároló elárvult …
|
||||
```
|
||||
|
||||
`repo_state: "orphaned"`, `orphaned_at: 2026-08-04T12:37:30Z`, `last_status: "error"`, the orphan card
|
||||
rendered on `/backups/remote`, and the event reached the hub (HTTP 200).
|
||||
|
||||
> **The system stopped and said so. It did not quietly start a new history over the old one.**
|
||||
> That closes R-193's open Q3 — and it is the good half of this month's story.
|
||||
|
||||
**Why the repository was orphaned:** the 15 snapshots / 40.9 MB at `/home/felhom-repo` were written
|
||||
under key `8e03eddf9ff7…` before the 2026-08-03 rebuild. That key survives only in superseded escrow
|
||||
row **id 3**, which carries **`identity_blob` = NULL** — it was superseded at `2026-08-04 07:15:36`,
|
||||
**four hours before** hub v0.93.0 fixed the retention. Unrecoverable, permanently, with or without a
|
||||
recovery code.
|
||||
|
||||
### Step 1c — the reset (operator-authorised)
|
||||
|
||||
The orphan card's own reset, confirmed by the operator during the session:
|
||||
|
||||
```
|
||||
[WARN] [offbox] resetting orphaned repo (operator-confirmed (claimed)):
|
||||
move-aside /home/felhom-repo -> /home/felhom-repo.orphaned-20260804, then re-init
|
||||
[INFO] [offbox] orphaned repo reset complete — old history set aside at
|
||||
/home/felhom-repo.orphaned-20260804 (move-aside, not deleted); fresh repo initialized
|
||||
[INFO] Event pushed: offbox_repo_reset (info)
|
||||
```
|
||||
|
||||
**Nothing was deleted.** The reset path had never run in anger before; it works.
|
||||
|
||||
### Steps 2–3 — the recovery code and the sentinel
|
||||
|
||||
The operator ran the ceremonies for **both** boxes earlier the same day and saved the codes. Measured
|
||||
hub-side: the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z`
|
||||
(demo-felhom), and `restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated
|
||||
them as same-password re-ceremonies: **no superseded row, no `offsite_repo_key_changed`**. That is
|
||||
v0.93.0's Scenario E, live.
|
||||
|
||||
**A file-leg app had to be deployed**, and this is a precondition the runbook did not anticipate:
|
||||
neither off-site-toggled app on the box (`opengist`, `privatebin`) has a restorable file leg — both
|
||||
keep everything in **named volumes**, which the off-site tier backs up as tars but the customer restore
|
||||
flow **never unpacks** (`offbox_reconstitute.go`). A sentinel in either would have been unrecoverable
|
||||
by design.
|
||||
|
||||
`calibre-web` was chosen: it declares `userdata: media/books class: mandatory`, and it is a
|
||||
single-container app. Deployed through the real API (`POST /api/stacks/calibre-web/deploy`, HTTP 202),
|
||||
toggled for off-site, and a Tier-1 recovery unit captured
|
||||
(`Recovery unit captured for calibre-web → …/backups/primary/calibre-web`).
|
||||
|
||||
Sentinel written, `sync`ed, hashed: **`643166269103a25c…`**, 181 B.
|
||||
|
||||
### Step 4 — the pre-wipe off-site backup: **`ok`, and wrong**
|
||||
|
||||
```
|
||||
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
|
||||
snapshot_count 3 · repo_size_bytes 30 636 · repo_state null
|
||||
```
|
||||
|
||||
Three snapshots, three apps, 37 s — and **the sentinel is in none of them**, per §2.
|
||||
|
||||
### Steps 5–11 — NOT RUN
|
||||
|
||||
Step 5 (verify the pre-wipe snapshot contents) would have confirmed the absence a second way; it is
|
||||
moot given §2. Steps 6–11 (archive, wipe, reinstall, recover, install, restore, compare) were **not
|
||||
started**. The §7 STOP was never reached, because the drill failed its own precondition first.
|
||||
|
||||
---
|
||||
|
||||
## 5. Part 0 — shipped, tested, deployed (R-200's plumbing half)
|
||||
|
||||
`--recover-offsite-install` (controller **v0.196.0**, commit `1b1366b`): same fetch → unseal → extract
|
||||
path as `--recover-offsite-check`, same STDIN discipline for R, but it **places** the recovered
|
||||
password via `InjectOffboxPassword`.
|
||||
|
||||
- **Confirmation is a second invocation.** Without `--confirm-install` it prints both hashes and writes
|
||||
nothing. A single interactive prompt would have had to share stdin with R.
|
||||
- **Three outcomes, named distinctly:** *installed* (no local password — the rebuilt-box shape),
|
||||
*unchanged* (identical key already present, nothing written), *refused* (a DIFFERENT key present;
|
||||
installing would clobber the key the current repository is encrypted under — exit 2, no force offered).
|
||||
- It re-reads the file after writing rather than trusting the call's return.
|
||||
|
||||
**Tests + red-proof.** `go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK.
|
||||
Removing the confirmation gate makes the dry run write the password and fails
|
||||
`TestRecoverAndInstall_InstallsOnABareBox` with *"the DRY RUN wrote the password"*. The R-persistence
|
||||
test carries a **positive control** — a planted copy of the code is found by the sweep, then removed
|
||||
and not found — because an absence check is worth only what its sensitivity is.
|
||||
|
||||
**It was deployed to demo-hp and never exercised against a live recovery**, because the drill halted
|
||||
before step 9. Its unit proof stands; its live proof does not exist.
|
||||
|
||||
---
|
||||
|
||||
## 6. What this drill did and did not establish
|
||||
|
||||
**Established, live, for the first time:**
|
||||
|
||||
1. A rebuilt box's off-site repository is **orphaned and the run refuses** — the spike's predicted
|
||||
third outcome, measured. It does not silently start a fresh history.
|
||||
2. The **orphan reset works**: move-aside, never delete, fresh repo initialised, event pushed.
|
||||
3. **A customer-declared mandatory data directory can be silently absent from the off-site snapshot
|
||||
while the run reports `ok`** (R-203).
|
||||
4. demo-hp's pre-rebuild off-site history is **permanently unrecoverable** — its key was destroyed four
|
||||
hours before the fix that would have kept it.
|
||||
|
||||
**NOT established — and unchanged from before this session:**
|
||||
|
||||
- **No file has ever been restored from an off-site backup after a wipe.** R-201's question is still
|
||||
open, and its pass condition is unchanged.
|
||||
- Part 0's install path has never run against a live recovery.
|
||||
- The v0.93.0 `identity_blob` retention is **still unit-proven only** — nothing in this session
|
||||
superseded a key, so nothing exercised it.
|
||||
|
||||
---
|
||||
|
||||
## 7. State left behind, and teardown
|
||||
|
||||
**Deliberately not torn down** — this is evidence, and the box is better off than it was:
|
||||
|
||||
| layer | state |
|
||||
|---|---|
|
||||
| the guest | `calibre-web` deployed and running, off-site-toggled, sentinel file in place. **Kept** as the drill fixture for the resumed run — it is the only app on either demo box with a restorable file leg. |
|
||||
| the off-site repository | fresh, working, 3 snapshots, 30 636 B. The old 15-snapshot history is **set aside**, not deleted, at `/home/felhom-repo.orphaned-20260804`. |
|
||||
| the host | agent v0.125.0, controller v0.196.0, `pvesm` unchanged apart from normal usage. |
|
||||
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` events recorded for `demo-hp`. No scratch customer records were created — **nothing was reinstalled**. |
|
||||
|
||||
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — that act is ruled and still owed,
|
||||
and §8.3 of the runbook forbids riding it along with a drill. The reset added
|
||||
`/home/felhom-repo.orphaned-20260804` (≈41 MB) to what is set aside.
|
||||
|
||||
**No `R` persisted anywhere** — the recovery code was never used in this session. It was read only to
|
||||
confirm the key exists in the credentials file; no unseal was performed on demo-hp.
|
||||
|
||||
---
|
||||
|
||||
## 8. To resume the drill
|
||||
|
||||
1. **Fix or scope R-203.** Until the mandatory userdata path lands in the snapshot, no sentinel can
|
||||
survive a wipe and the drill cannot reach its verdict.
|
||||
2. Re-run steps 4–5 and confirm the sentinel IS in the snapshot — by listing it, not by a green status.
|
||||
3. Then P4 (the deliberate archive), the §7 STOP, and steps 6–11 as written.
|
||||
|
||||
Everything else is already in place: the code, the versions, the recovery code, the working repository,
|
||||
the file-leg app and the sentinel.
|
||||
@@ -75,9 +75,10 @@ file with it.**
|
||||
| **R-197** | **The hub holds both halves of the evidence that a box's offsite DATA key changed, and reads neither.** `restic_pw_sha256` is stored on `host_escrow` and carried to `host_escrow_superseded` on every re-escrow. Comparing the two is what let this spike answer its hardest question in one query — and **nothing in the hub does it** | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **Why this is the cheapest real fix on the table.** A changed repo password means the previous offsite repository is now unopenable by the box, i.e. the customer's off-site history is orphaned. That is the single most consequential state change in the backup system, it is **already fully observable from data the hub owns**, and today it produces **no event, no e-mail, no card and no log line** — demo-felhom's went unremarked for 13 hours and would have gone unremarked indefinitely had this spike not run. **Two-line verdict:** on `SaveHostEscrow`, if the incoming `restic_pw_sha256` differs from the row being superseded, emit a distinct operator event naming the orphaned generation. **Deliberately an EVENT, not a heal** — nothing should act on this automatically until R-193's (c)-vs-accept decision is taken; the point is that the operator learns on the day. **Pair with R-196**, which is the same signal aimed at the right trigger. **Generalises past this row:** *a comparison the system could be making from data it already stores, and is not, is a silence with no cost of entry* — cf. R-190's store-grant probe, where the state was read and the TRANSITION was not. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q8 option (d) **SHIPPED.** `SaveHostEscrow` returns the hash it replaced; `handleHostEscrowPut` raises **`offsite_repo_key_changed`** when both hashes are known and differ. **Edge-triggered** (once per supersession, never per report — the dispatcher owns cooldown), **operator-only** (registered in `notify.operatorOnlyEvents` in the same commit that mints the type, because a missing `customerMessages` entry is NOT a block — the v0.78.0 defect), and **no hash value travels** in the message or the details. The in-between shapes (a first-ever hash, a hash-less supersession) are LOGGED rather than dropped, so *"we chose not to alarm"* and *"the check did not run"* never look identical. **Severity = warning, chosen for the world v0.93.0 creates:** before R-198 a changed key meant the previous history was unopenable by anyone ever, which would have argued for `error`; from v0.93.0 the superseding ceremony retains the old identity blob, so the fact is *"this customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss. `warning` also routes (the dispatcher treats `info` as an intentional non-notify). Driven through the real endpoint in test, not by calling the emitter. **Red-proof observed:** removing the comparison from the escrow PUT → the changed-key scenario fails with *"the repository key demonstrably changed and NO signal was raised"* while the unchanged-key scenario still passes. | CC |
|
||||
| **R-198** | **The hub's superseded-escrow retention does NOT retain the offsite repository password — and the ceremony the system tells the customer to run is what destroys the last copy.** `host_escrow_superseded` has **no `identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) copies only `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256`. `blob` is the **K-escrow** (the PBS datastore key, PBS-native scrypt); the **restic repo password lives in `identity_blob`** (`felhom-agent/internal/escrow/identity.go:34-39`, age-wrapped `IdentityBundle`). Measured live: both hosts' current rows hold `blob`=383 B **and** `identity_blob`=572 B; both superseded rows hold `blob`=383 B and nothing else | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **This is the NINTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does not provide, and the first that is ALSO customer-facing copy.** The claim appears three times: the schema comment (`store.go:370-375`, *"so the old passphrase stays customer-R-recoverable … turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable'"*), the capability map's escrow row, and — in Hungarian, to the customer, on the orphan card — `controller/internal/web/templates/backups_remote.html:66,69` (*„a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók lehetnek"*). **For the offsite restic repository, the incident it names, all three are false.** **Why it is worse than a missing column:** a rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`, `offbox.go:570`), the card says „Helyreállítási kód szükséges", and the controller's own detector logs *"run the escrow ceremony"* (`report/escrow_confirm.go:100`) — so **the prescribed remedy is the act that overwrites `host_escrow.identity_blob` and loses the old password forever**. Both demo boxes crossed that line on **2026-08-04 at 07:15:36 (demo-hp) and 07:20:08 (demo-felhom)**. **This is an independent, stronger reason the 51 orphaned demo snapshots are unrecoverable than "nobody kept the recovery codes" — keeping R would not have helped.** **Fix shape (small):** add `identity_blob` to the superseded table and to `demoteCurrentEscrowTx`'s SELECT list; pin it with a test that asserts the CONSEQUENCE (a superseded row can still yield a repo password) rather than the mechanism. **Then correct all three claims in the same commit** — including the Hungarian card, which must not promise what the system cannot do. **Prerequisite for R-193's (d)-alone branch** and for the drill. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §7 **SHIPPED.** `host_escrow_superseded` gains `identity_blob` (CREATE + additive `ALTER TABLE`) and `demoteCurrentEscrowTx` carries it, so **both** callers — re-escrow and host-delete demotion — are fixed by one change to the routine its own comment calls *"THE ONE escrow row-copy routine"*. `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob is reachable from Go at all. The table comment now records that the ruling stated above it was not met and what that cost. **Tests assert the CONSEQUENCE, which is why the existing one stayed green:** `TestSaveHostEscrow_RetainsSuperseded` asserted that a retained row exists carrying the old K-blob and passed throughout; `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still yield a repository password, and pins the load-bearing ordering (the identity blob is written by `SaveHostDRBundle` AFTER `SaveHostEscrow`, so the demote sees the PREVIOUS generation — if that inverts, the retained bytes would be the new blob filed under the old hash, recoverable-looking and wrong). `TestDeleteHost_DemotesIdentityBlob` proves the shared routine through its other caller. **Red-proofs, both observed failing:** dropping `identity_blob` from the copy (production behaviour ≤ v0.92.0) fails BOTH scenarios; fixing only the re-escrow caller fails the delete scenario while the re-escrow one passes — the §8.2 mistake, demonstrated rather than asserted. **Nothing was backfillable and it was CHECKED, not deduced:** rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; the live database holds exactly 2 retained rows, both from 2026-08-04, both `identity_blob` NULL. **Who the fix protects, measured live:** 2 of 2 hosts with a current escrow carry an identity blob (`demo-felhom-8363b5`, `demo-hp-bb76ea`) — their NEXT ceremony now retains a recoverable off-site key instead of destroying one. | CC |
|
||||
| **R-199** | **The hub serves recovery blobs on two endpoints that have no client anywhere in the system.** `handleReEnroll` (`hub/internal/api/dr.go:101`) and `handleGetRestoreDirective` (`:155`) return `identity_escrow_b64` + `k_escrow_b64`, gated on operator-armed recovery mode. Census: **zero** callers in `felhom-agent` (no `ReEnroll` symbol at all; `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`), **zero** in the hub UI or any template, **zero** in `scripts/` or any runbook | **SHIPPED + PROVEN-LIVE 2026-08-04** (hub **v0.94.0**, agent **v0.125.0**) | — | **The documented retrieval path is a human with `sqlite3`:** `SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …` on a `kubectl cp`-ed `hub.db` — recorded in project memory as the 2026-07-04 S5 prep, where hub-side blob serving was explicitly deemed *"Part 3 NOT needed"*. **This is the built-but-never-wired class at the DR capstone**, and it is why the chain from a dead node to an open repository has no automatable middle. **What a recovery flow actually needs is smaller than what exists:** a narrow `GET /hosts/<id>/escrow` authed with the box's own per-host key, serving opaque bytes to the box that owns them — zero-knowledge untouched, and far lighter than `re-enroll`, which **rotates the host API key** and returns the new key in the response body (`dr.go:130,148`). **Decide before building:** whether `re-enroll`/`restore-directive` should get a client, be replaced by the narrow GET, or be retired. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 6 **LINKS 6, 7 AND 8 ARE ASSEMBLED AND WALKED.** **Link 6:** `GET /api/v1/hosts/{host_id}/escrow` — the box-authenticated MIRROR of the PUT that stores the blob, self-scoped by the per-host key (global may read any, the same asymmetry the PUT has). The operator-driven DR endpoints are UNTOUCHED and pinned by a test that exercises them with recovery mode off and on. **Link 7:** `POST /escrow/recover-offsite-password` on the agent's pinned local API gives `UnwrapIdentityBundle` its first production caller in two months. **Link 8:** it extracts and returns **only** the repository password (not the tunnel token, not the PBS token, not the WG key — the controller is a trust tier down). **PROVEN ON HARDWARE, demo-felhom, 2026-08-04 13:49 CEST:** on-disk `c60c8bc737a6…` vs recovered `c60c8bc737a6…` — **MATCH**, and the same hash the hub independently stores as `restic_pw_sha256`, so three sources agree. **Scenario B proven live 5 minutes earlier** with a deliberately wrong code: hub served the blob (572 B, `self_scope=true`), agent logged *the recovery code did not unwrap the identity escrow … exit status 1*, nothing written — which also proves links 6 and 7 ran independently of the success. **Scenario E proven live:** both retrievals raised `escrow_blob_served` (warning, operator-only); the first mailed the operator, the second was cooldown-suppressed AND that suppression is itself recorded; the customer leg reads `skipped/operator_only` on both. **R persisted nowhere, searched not claimed:** 0 lines in the agent journal, 0 in the controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover `felhom-idesc-*` staging dirs, and the staged-secret dir empty — with a **positive control** (a planted copy was found: 1, then 0 after removal) so the sweep is a measurement rather than an unfalsifiable absence. **THE §8.2 TRADE, made deliberately and recorded in the handler:** obtaining the blob used to require the operator to arm recovery mode; it now needs only the box's own credential. They still cannot open it (the hub never held R; a wrong code fails closed at age's scrypt KDF). `escrowSelfServiceRetrieval` is a single named constant — flipping it to false re-imposes recovery mode and changes nothing else, so the operator can overrule the trade for the cost of a boolean. **Red-proofs observed:** removing the ownership check served host B's blob to host A; removing the audit record made the retrieval silent; returning `PBSToken` instead of `ResticRepoPassword` yielded a plausible bundle with a non-matching key; commenting the `Options.EscrowRecovery` wiring failed the AST seam test. **Seam discipline:** the wiring is asserted by walking `main` → `runDaemon` → `buildLocalAPIServer` and checking the composite literal, not by `strings.Contains` — links 6 and 7 were two of this project's six built-but-never-wired instances and the fix must not become the seventh | CC |
|
||||
| **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **PLUMBING SHIPPED** (controller **v0.195.0**); **the FORM is not built — still open** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 **THE DIAGNOSTIC HALF IS DONE AND IT ANSWERED THE QUESTION.** `--recover-offsite-check` is a `docker exec` escape hatch in the shape of `--print-reset-code`: R on **STDIN** (never argv, never `ps`, never shell history, never a transcript), fetch+unseal via the agent, and a verdict of **two sha256 hashes**. **It compares and never installs** — the recovered password is not written to `offbox/repo_password`; a test asserts the data dir is byte-unchanged and its red-proof (adding the install call) fails it. Confirmed live: `repo_password` mtime still `2026-08-03 07:18:02` after the successful check at `2026-08-04 11:49`. **Exit codes are load-bearing** — `0` match, `2` a clean MISMATCH, `1` a step failed; "it failed" and "it worked and disagreed" must never share a status because only one is a finding about the system. A box with no local password reports distinctly (the rebuilt-box shape, where the next step is to INSTALL rather than compare). **WHAT IS NOT BUILT, deliberately:** no card, no form, no preview, no customer-facing text — building an interface on top of a chain nobody had walked is how the preceding three weeks went wrong. **What remains for this row:** link 9 (the recovered password placed so `WriteOffboxSecrets` keeps it) and the customer-facing shape the operator ruled on 2026-08-04 (yell → R form → preview → proceed), which is now priced against a chain that exists rather than one that is assumed | CC |
|
||||
| **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **OPEN — the KEY is proven recoverable; the RESTORE is not** | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there **MATERIALLY ADVANCED 2026-08-04, AND THE DISTINCTION MATTERS.** What was proven on hardware is that the offsite repository **password** comes back out of the sealed bundle byte-identical (R-199). What has STILL never happened: a recovered password **installed**, an existing repository **reopened** under one, and a **file restored** from it. **The drill's pass condition is unchanged and is not this** — it is a byte-identical sha256 of a sentinel file after a wipe and reinstall, and "the store opened" was explicitly ruled insufficient. Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10. **The drill is now cheaper and better-founded than when it was designed:** links 6–8 are walked, so a failure during the drill can be localised instead of being an undifferentiated "recovery did not work", and R-198 means the ceremony that creates the drill's recovery code no longer destroys the key it is meant to protect | CC + operator |
|
||||
| **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **PLUMBING COMPLETE** (controller **v0.196.0**); **the FORM is not built — still open** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 **THE DIAGNOSTIC HALF IS DONE AND IT ANSWERED THE QUESTION.** `--recover-offsite-check` is a `docker exec` escape hatch in the shape of `--print-reset-code`: R on **STDIN** (never argv, never `ps`, never shell history, never a transcript), fetch+unseal via the agent, and a verdict of **two sha256 hashes**. **It compares and never installs** — the recovered password is not written to `offbox/repo_password`; a test asserts the data dir is byte-unchanged and its red-proof (adding the install call) fails it. Confirmed live: `repo_password` mtime still `2026-08-03 07:18:02` after the successful check at `2026-08-04 11:49`. **Exit codes are load-bearing** — `0` match, `2` a clean MISMATCH, `1` a step failed; "it failed" and "it worked and disagreed" must never share a status because only one is a finding about the system. A box with no local password reports distinctly (the rebuilt-box shape, where the next step is to INSTALL rather than compare). **WHAT IS NOT BUILT, deliberately:** no card, no form, no preview, no customer-facing text — building an interface on top of a chain nobody had walked is how the preceding three weeks went wrong. **What remains for this row:** link 9 (the recovered password placed so `WriteOffboxSecrets` keeps it) and the customer-facing shape the operator ruled on 2026-08-04 (yell → R form → preview → proceed), which is now priced against a chain that exists rather than one that is assumed **PART 0 SHIPPED 2026-08-04 (v0.196.0):** `--recover-offsite-install` is the sibling of the check — same fetch/unseal path, same STDIN discipline for R — and it **places** the recovered password via `InjectOffboxPassword`. The confirmation is a SECOND invocation (`--confirm-install`): without it, both hashes print and nothing is written, so the operator sees the comparison before a write is possible. Three outcomes named distinctly: **installed** (no local password — the rebuilt-box shape), **unchanged** (identical key, nothing written), **refused** (a DIFFERENT key present — installing would clobber the key the current repository is encrypted under; exit 2, no force offered). It re-reads the file after writing rather than trusting the call. **Red-proof observed:** removing the confirmation gate makes the dry run write the password. The R-persistence test carries a **positive control** (a planted copy found, then removed and not found). **NOT YET EXERCISED AGAINST A LIVE RECOVERY** — the R-201 drill halted before step 9, so this is unit-proven only. **What remains for this row:** the customer-facing shape the operator ruled on 2026-08-04 (yell → recovery-code form → preview → proceed) | CC |
|
||||
| **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **OPEN — the drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04)** | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there **MATERIALLY ADVANCED 2026-08-04, AND THE DISTINCTION MATTERS.** What was proven on hardware is that the offsite repository **password** comes back out of the sealed bundle byte-identical (R-199). What has STILL never happened: a recovered password **installed**, an existing repository **reopened** under one, and a **file restored** from it. **The drill's pass condition is unchanged and is not this** — it is a byte-identical sha256 of a sentinel file after a wipe and reinstall, and "the store opened" was explicitly ruled insufficient. Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10. **The drill is now cheaper and better-founded than when it was designed:** links 6–8 are walked, so a failure during the drill can be localised instead of being an undifferentiated "recovery did not work", and R-198 means the ceremony that creates the drill's recovery code no longer destroys the key it is meant to protect **THE DRILL RAN AND DID NOT REACH ITS VERDICT, and stopping was the correct call** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). Steps 1–4 completed; the wipe never happened; **nothing irreversible was done**. It halted because **the sentinel file was not in the off-site snapshot** (R-203) — wiping would have destroyed the only copy and proven nothing. **Sentinel sha256 `643166269103a25c…`, still on the box.** **What the attempt established live, all of it new:** (1) a rebuilt box's off-site run **refuses** with the orphan card and pushes `offbox_repo_orphaned` — the spike's predicted third outcome, measured for the first time, and it does NOT silently start a fresh history (this closes R-193's Q3); (2) the **orphan reset works** — move-aside to `/home/felhom-repo.orphaned-20260804`, never delete, fresh repo initialised, `offbox_repo_reset` pushed (operator-authorised during the session); (3) demo-hp's pre-rebuild history is **permanently unrecoverable** — its key sits in superseded row id 3 with `identity_blob` NULL, superseded 07:15:36, **four hours before v0.93.0 fixed the retention**; (4) **a precondition the runbook did not contain:** neither pre-existing off-site-toggled app has a restorable file leg — both are named-volume-only, which the off-site tier tars but the customer restore never unpacks — so a file-leg app (`calibre-web`, mandatory `userdata: media/books`) had to be deployed, and it is now in place as the fixture. **TO RESUME:** fix or scope R-203, re-run steps 4–5 and confirm the sentinel IS in the snapshot **by listing it, not by a green status**, then P4 + the STOP + steps 6–11. Everything else is already staged: versions, recovery code, working repository, file-leg app, sentinel. **The pass condition is unchanged** — a byte-identical sentinel sha256, not "the store opened" | CC + operator |
|
||||
| **R-202** | **The orphan card promises the customer their old backups "may later be restorable with the matching recovery code" — and after v0.93.0 that is true for supersessions from now on and FALSE for anything already orphaned.** `controller/internal/web/templates/backups_remote.html:66,69` states it unconditionally, in Hungarian, on the one surface where being wrong costs most | **OPEN — Part 5 hit its gate 2026-08-04; the card is UNTOUCHED and the sentence is still live** | R-199/R-201 (which generation an orphaned repo belongs to is not knowable to the box today) | — | **THE GATE, and why it was hit rather than squeezed past.** The condition was: ship it iff the hub can tell a box what it needs with **one** additional boolean on the escrow ACK it already sends. The hub *can* cheaply compute *"≥1 retained blob for this host carries an identity blob"* — one correlated predicate in `GetEscrowStatusForCustomer`, and the controller even has the right seam already (`SetEscrowStale`/`StaleBlob` is exactly this shape). **But that boolean does not answer the card's question.** The card renders on `RepoState == "orphaned"`, and the promise is about *the key THIS orphaned repository was written under*. A box does not know which escrow generation the orphaned remote belongs to; a box with a pre-v0.93.0 orphan and a post-v0.93.0 supersession would read the boolean TRUE and the promise would still be false — a **conditional** falsehood that looks verified, which is strictly worse on a customer-facing card than today's hedged one. **What would actually make it truthful** is knowing the orphaned repo's generation, which is the same knowledge R-199/R-201's unassembled chain needs. **Interim exposure, stated rather than buried:** the sentence remains live and remains false for both demo boxes. **Cheapest honest interim** (not taken here — it is a customer-copy change and the gate said leave it alone): drop the recoverability clause and say only that the old history is set aside and not deleted, which is true unconditionally | CC + operator |
|
||||
| **R-203** | **A customer-declared MANDATORY data directory was silently absent from the off-site snapshot while the run reported `ok`.** Measured live on demo-hp 2026-08-04: `calibre-web` declares `userdata: media/books class: mandatory`; its live bind is `/mnt/sys_drive/userdata/media/books` (the sentinel file was there), while the off-site capture set looked for `/mnt/sys_drive/felhom-data/userdata/media/books`, which does not exist. Result: `[WARN] mandatory data path missing on disk, skipped from offsite`, then `backed up calibre-web (…, 0 mandatory path(s))` and `backup OK: 3 app(s), 3 snapshot(s)` — `last_status: ok`, `last_success` stamped, nothing customer-visible, nothing hub-visible | **OPEN — HALTED THE R-201 DRILL** | — | **THE MECHANISM, from source.** `NamespaceRoot(drivePath, inGuestDrive)` (`appbackup/paths.go:28-33`) appends the `felhom-data` segment **when the drive IS the system data path** (`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`, `backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is `<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an app deployed at `HDD_PATH=/mnt/sys_drive`, the two produce different directories. **The same compose used BOTH roots**, from one deploy: `${IMPORT_PATH}` → `/mnt/sys_drive/felhom-data/userdata/import/calibre` (with the segment), `${USERDATA_PATH}` → `/mnt/sys_drive/userdata/media/books` (without). **WHAT IS MEASURED vs NOT, because it changes the fix.** MEASURED: the paths disagree, the mandatory directory is absent from the snapshot, the run says `ok`, and the only signal is a container-log WARN. **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a SUPPORTED choice — it was used because demo-hp's only registered drive (`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working as designed), while `/mnt/sys_drive` was **accepted** (HTTP 202). **EITHER BRANCH IS A DEFECT:** if the system drive is a supported app namespace, userdata resolution is wrong for every app on it and their mandatory directories are silently unprotected; if it is not supported, the deploy accepted a namespace it should have refused one call after refusing the NAS. **NOT a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths (everything of theirs is in named volumes), so they are unaffected and their snapshots are real. **Fix shape:** make the two roots one function, whichever is right — and make a skipped MANDATORY path a customer/hub-visible signal rather than a WARN, because `ok` with a missing mandatory directory is this project's own *a path the customer thinks is protected is not in the snapshot* shape. Source: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 | CC |
|
||||
| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **HALF SHIPPED** (hub **v0.93.0**) — **the guard's SCOPING stays OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** unLine truncated
|
||||
| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** — the (c) decision is TAKEN (accept the risk); the chain is still unassembled | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/esLine truncated
|
||||
| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
|
||||
|
||||
@@ -63,6 +63,8 @@
|
||||
| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
|
||||
| R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 6–8 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
|
||||
| R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
|
||||
| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
|
||||
| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
|
||||
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
|
||||
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
||||
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |
|
||||
|
||||
Reference in new issue
Block a user