312 lines
17 KiB
Markdown
312 lines
17 KiB
Markdown
# DRILL RECORD — R-201, the wipe-and-recover proof: **PREPARED, HALTED BEFORE THE WIPE**
|
||
|
||
**Date:** 2026-08-04 · **Box:** `demo-hp` (Tier 0, the designated drill host) · **Nothing was wiped.**
|
||
**Outcome:** the drill did not reach its verdict. It was halted at step 4 by a defect that makes the
|
||
verdict unobtainable — and that defect is worth more than the drill.
|
||
|
||
> **THE HEADLINE.** A customer-declared **mandatory** data directory was **silently absent from the
|
||
> off-site snapshot**, while the backup reported `ok` with three snapshots. The only trace is one
|
||
> `[WARN]` line inside the controller container. The hub, the card and the counters all say the backup
|
||
> succeeded. → **R-203**
|
||
>
|
||
> **Nothing irreversible happened.** The wipe never ran. `demo-hp` is left in a BETTER state than it
|
||
> started: it now has a working off-site repository (it had an orphaned one), three snapshots, and a
|
||
> sentinel file on disk.
|
||
|
||
---
|
||
|
||
## 1. The verdict — not reached, and why that is the correct outcome
|
||
|
||
The drill's pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established that
|
||
the sentinel is not in the off-site snapshot at all.** Wiping the box would therefore have:
|
||
|
||
- destroyed the sentinel, which exists only on that box;
|
||
- proven nothing about recovery, because there would be nothing to recover;
|
||
- and done so *after* the point of no return.
|
||
|
||
The runbook's own rule applies: *"If this session finds itself writing code beyond Part 0, stop. That
|
||
means a precondition was wrong, and the finding outranks the drill."* A precondition was wrong. It was
|
||
one the runbook's P1–P6 table did not contain, because nobody knew to look for it.
|
||
|
||
**Sentinel sha256 (step 3), recorded and still on the box:**
|
||
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
|
||
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt` (181 B).
|
||
|
||
---
|
||
|
||
## 2. R-203 — the defect that halted the drill
|
||
|
||
**Measured, twice, on the live box.**
|
||
|
||
| what | path | exists? |
|
||
|---|---|---|
|
||
| the app's live bind, where the customer's files actually land | `/mnt/sys_drive/userdata/media/books` | **YES** (the sentinel is here) |
|
||
| the path the off-site capture set treats as the mandatory directory | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
|
||
|
||
The controller's own log, verbatim:
|
||
|
||
```
|
||
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
|
||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||
[INFO] [offbox] backed up calibre-web (…/backups/primary/calibre-web, 0 mandatory path(s))
|
||
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
|
||
```
|
||
|
||
**The run reported `ok`.** `last_status: ok`, `last_success` stamped, `snapshot_count: 3`. Nothing
|
||
customer-visible, nothing hub-visible and nothing in any counter says the mandatory directory was
|
||
dropped. This is `CLAUDE.md`'s recurring shape — *a path the customer thinks is protected is not in the
|
||
snapshot* — and the code even has the right words for it in a WARN nobody reads.
|
||
|
||
**The mechanism, from source.** `NamespaceRoot(drivePath, inGuestDrive)`
|
||
(`appbackup/paths.go:28-33`) appends `felhom-data` **when the drive IS the system data path**
|
||
(`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`,
|
||
`backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is documented and computed
|
||
as **`<HDD_PATH>/userdata`** (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive`
|
||
and an app deployed at `HDD_PATH=/mnt/sys_drive`, those two produce different directories.
|
||
|
||
**And the same compose file used BOTH roots.** From `docker inspect calibre-web`:
|
||
|
||
```
|
||
bind /mnt/sys_drive/felhom-data/userdata/import/calibre -> /cwa-book-ingest ← felhom-data root
|
||
bind /mnt/sys_drive/userdata/media/books -> /calibre-library ← NO felhom-data root
|
||
```
|
||
|
||
`${IMPORT_PATH}` resolved *with* the segment; `${USERDATA_PATH}` resolved *without* it. One deploy, one
|
||
template, two roots.
|
||
|
||
### What is measured and what is not — stated because the scope changes the fix
|
||
|
||
- **MEASURED:** the two paths disagree; the mandatory directory is absent from the snapshot; the run
|
||
reports `ok`; the only signal is a container-log WARN.
|
||
- **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a *supported* choice. It was chosen because
|
||
demo-hp's only registered drive, `Felhom-Share`, is a NAS and was **correctly refused** as an app
|
||
namespace (R-108's `RefuseAsAppNamespace`, working as designed). `/mnt/sys_drive` was **accepted**
|
||
(HTTP 202) rather than refused.
|
||
|
||
**Either branch is a defect, which is why this is filed regardless:**
|
||
|
||
- if the system drive **is** a supported app namespace → the userdata path resolution is wrong for
|
||
every app deployed on it, and their mandatory directories are silently unprotected;
|
||
- if it is **not** supported → the deploy accepted a namespace it should have refused, exactly as it
|
||
refused the NAS one call earlier, and the refusal that exists is not reaching this case.
|
||
|
||
**What must NOT be concluded from this drill:** that off-site backups are broken generally. The two
|
||
pre-existing apps (`opengist`, `privatebin`) declare **no** mandatory userdata paths — everything they
|
||
own is in named volumes — so they are unaffected, and their snapshots are real.
|
||
|
||
---
|
||
|
||
## 3. Preconditions, each measured
|
||
|
||
| # | Precondition | Result |
|
||
|---|---|---|
|
||
| **P1** | operator holds the recovery code | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file (`~/.config/credentials`). *The code itself appears nowhere in this record.* |
|
||
| **P2** | demo-hp on agent v0.125.0 + controller v0.196.0 | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed the published v0.125.0 (sha `f7d8339b53d9…`, verified against the release) and controller v0.196.0. `age` present at `/usr/bin/age`. |
|
||
| **P3** | the current `identity_blob` seals the repository under test | **PASS with a caveat that reshaped the drill** — `identity_blob` = 572 B, `restic_pw_sha256` = `8a9e33aa4da6…`. But **no repository existed under that key** (see §4). |
|
||
| **P4** | a deliberate, verified whole-guest archive as rollback | **NOT TAKEN** — deliberately. It is only needed for the wipe, and the wipe did not happen. |
|
||
| **P5** | demo-felhom untouched and healthy | **PASS** — not touched at any point. |
|
||
| **P6** | space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 % used, guest 64 GB free. |
|
||
|
||
---
|
||
|
||
## 4. Step-by-step, with every observable
|
||
|
||
### Step 1 — starting state (hub, read-only)
|
||
|
||
`demo-hp-bb76ea`: `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, created `2026-08-04T11:11:37Z`, `escrow_state: escrowed`.
|
||
Report: `snapshot_count: 0`, `repo_size_bytes: 0`, **no `last_run`/`last_status` at all** — the shape of
|
||
a controller that has never run an off-site backup in this lifetime.
|
||
|
||
### Step 1b — the repository was ORPHANED, and this is the first live proof of a prediction
|
||
|
||
The 2026-08-04 spike predicted from source — and could not measure — that a rebuilt box's next
|
||
off-site run would hit a **third** outcome: neither reattaching the old snapshots nor silently starting
|
||
a fresh history, but **refusing**. Measured here, twice over.
|
||
|
||
**Read-only probe first** (`restic cat config` with the current key, writes nothing):
|
||
|
||
```
|
||
Fatal: wrong password or no key found
|
||
```
|
||
|
||
— the exact string `classifyResticProbe` maps to `"orphaned"`.
|
||
|
||
**Then the real run, through the customer UI endpoint** (`POST /backup/offbox/run`):
|
||
|
||
```
|
||
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
|
||
no-longer-available key; runs will skip until reset
|
||
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
|
||
[INFO] Event pushed: offbox_repo_orphaned (warning) — A távoli mentési tároló elárvult …
|
||
```
|
||
|
||
`repo_state: "orphaned"`, `orphaned_at: 2026-08-04T12:37:30Z`, `last_status: "error"`, the orphan card
|
||
rendered on `/backups/remote`, and the event reached the hub (HTTP 200).
|
||
|
||
> **The system stopped and said so. It did not quietly start a new history over the old one.**
|
||
> That closes R-193's open Q3 — and it is the good half of this month's story.
|
||
|
||
**Why the repository was orphaned:** the 15 snapshots / 40.9 MB at `/home/felhom-repo` were written
|
||
under key `8e03eddf9ff7…` before the 2026-08-03 rebuild. That key survives only in superseded escrow
|
||
row **id 3**, which carries **`identity_blob` = NULL** — it was superseded at `2026-08-04 07:15:36`,
|
||
**four hours before** hub v0.93.0 fixed the retention. Unrecoverable, permanently, with or without a
|
||
recovery code.
|
||
|
||
### Step 1c — the reset (operator-authorised)
|
||
|
||
The orphan card's own reset, confirmed by the operator during the session:
|
||
|
||
```
|
||
[WARN] [offbox] resetting orphaned repo (operator-confirmed (claimed)):
|
||
move-aside /home/felhom-repo -> /home/felhom-repo.orphaned-20260804, then re-init
|
||
[INFO] [offbox] orphaned repo reset complete — old history set aside at
|
||
/home/felhom-repo.orphaned-20260804 (move-aside, not deleted); fresh repo initialized
|
||
[INFO] Event pushed: offbox_repo_reset (info)
|
||
```
|
||
|
||
**Nothing was deleted.** The reset path had never run in anger before; it works.
|
||
|
||
### Steps 2–3 — the recovery code and the sentinel
|
||
|
||
The operator ran the ceremonies for **both** boxes earlier the same day and saved the codes. Measured
|
||
hub-side: the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z`
|
||
(demo-felhom), and `restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated
|
||
them as same-password re-ceremonies: **no superseded row, no `offsite_repo_key_changed`**. That is
|
||
v0.93.0's Scenario E, live.
|
||
|
||
**A file-leg app had to be deployed**, and this is a precondition the runbook did not anticipate:
|
||
neither off-site-toggled app on the box (`opengist`, `privatebin`) has a restorable file leg — both
|
||
keep everything in **named volumes**, which the off-site tier backs up as tars but the customer restore
|
||
flow **never unpacks** (`offbox_reconstitute.go`). A sentinel in either would have been unrecoverable
|
||
by design.
|
||
|
||
`calibre-web` was chosen: it declares `userdata: media/books class: mandatory`, and it is a
|
||
single-container app. Deployed through the real API (`POST /api/stacks/calibre-web/deploy`, HTTP 202),
|
||
toggled for off-site, and a Tier-1 recovery unit captured
|
||
(`Recovery unit captured for calibre-web → …/backups/primary/calibre-web`).
|
||
|
||
Sentinel written, `sync`ed, hashed: **`643166269103a25c…`**, 181 B.
|
||
|
||
### Step 4 — the pre-wipe off-site backup: **`ok`, and wrong**
|
||
|
||
```
|
||
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
|
||
snapshot_count 3 · repo_size_bytes 30 636 · repo_state null
|
||
```
|
||
|
||
Three snapshots, three apps, 37 s — and **the sentinel is in none of them**, per §2.
|
||
|
||
### Steps 5–11 — NOT RUN
|
||
|
||
Step 5 (verify the pre-wipe snapshot contents) would have confirmed the absence a second way; it is
|
||
moot given §2. Steps 6–11 (archive, wipe, reinstall, recover, install, restore, compare) were **not
|
||
started**. The §7 STOP was never reached, because the drill failed its own precondition first.
|
||
|
||
---
|
||
|
||
## 5. Part 0 — shipped, tested, deployed (R-200's plumbing half)
|
||
|
||
`--recover-offsite-install` (controller **v0.196.0**, commit `1b1366b`): same fetch → unseal → extract
|
||
path as `--recover-offsite-check`, same STDIN discipline for R, but it **places** the recovered
|
||
password via `InjectOffboxPassword`.
|
||
|
||
- **Confirmation is a second invocation.** Without `--confirm-install` it prints both hashes and writes
|
||
nothing. A single interactive prompt would have had to share stdin with R.
|
||
- **Three outcomes, named distinctly:** *installed* (no local password — the rebuilt-box shape),
|
||
*unchanged* (identical key already present, nothing written), *refused* (a DIFFERENT key present;
|
||
installing would clobber the key the current repository is encrypted under — exit 2, no force offered).
|
||
- It re-reads the file after writing rather than trusting the call's return.
|
||
|
||
**Tests + red-proof.** `go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK.
|
||
Removing the confirmation gate makes the dry run write the password and fails
|
||
`TestRecoverAndInstall_InstallsOnABareBox` with *"the DRY RUN wrote the password"*. The R-persistence
|
||
test carries a **positive control** — a planted copy of the code is found by the sweep, then removed
|
||
and not found — because an absence check is worth only what its sensitivity is.
|
||
|
||
**It was deployed to demo-hp and never exercised against a live recovery**, because the drill halted
|
||
before step 9. Its unit proof stands; its live proof does not exist.
|
||
|
||
---
|
||
|
||
## 6. What this drill did and did not establish
|
||
|
||
**Established, live, for the first time:**
|
||
|
||
1. A rebuilt box's off-site repository is **orphaned and the run refuses** — the spike's predicted
|
||
third outcome, measured. It does not silently start a fresh history.
|
||
2. The **orphan reset works**: move-aside, never delete, fresh repo initialised, event pushed.
|
||
3. **A customer-declared mandatory data directory can be silently absent from the off-site snapshot
|
||
while the run reports `ok`** (R-203).
|
||
4. demo-hp's pre-rebuild off-site history is **permanently unrecoverable** — its key was destroyed four
|
||
hours before the fix that would have kept it.
|
||
|
||
**NOT established — and unchanged from before this session:**
|
||
|
||
- **No file has ever been restored from an off-site backup after a wipe.** R-201's question is still
|
||
open, and its pass condition is unchanged.
|
||
- Part 0's install path has never run against a live recovery.
|
||
- The v0.93.0 `identity_blob` retention is **still unit-proven only** — nothing in this session
|
||
superseded a key, so nothing exercised it.
|
||
|
||
---
|
||
|
||
## 7. State left behind, and teardown
|
||
|
||
**Deliberately not torn down** — this is evidence, and the box is better off than it was:
|
||
|
||
| layer | state |
|
||
|---|---|
|
||
| the guest | `calibre-web` deployed and running, off-site-toggled, sentinel file in place. **Kept** as the drill fixture for the resumed run — it is the only app on either demo box with a restorable file leg. |
|
||
| the off-site repository | fresh, working, 3 snapshots, 30 636 B. The old 15-snapshot history is **set aside**, not deleted, at `/home/felhom-repo.orphaned-20260804`. |
|
||
| the host | agent v0.125.0, controller v0.196.0, `pvesm` unchanged apart from normal usage. |
|
||
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` events recorded for `demo-hp`. No scratch customer records were created — **nothing was reinstalled**. |
|
||
|
||
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — that act is ruled and still owed,
|
||
and §8.3 of the runbook forbids riding it along with a drill. The reset added
|
||
`/home/felhom-repo.orphaned-20260804` (≈41 MB) to what is set aside.
|
||
|
||
**No `R` persisted anywhere** — the recovery code was never used in this session. It was read only to
|
||
confirm the key exists in the credentials file; no unseal was performed on demo-hp.
|
||
|
||
---
|
||
|
||
## 7b. UPDATE, same day — the blocker is FIXED and the drill is ready to resume
|
||
|
||
Controller **v0.197.0** shipped both halves of R-203:
|
||
|
||
- **the paths agree.** On demo-hp the live bind moved from `/mnt/sys_drive/userdata/media/books` to
|
||
`/mnt/sys_drive/felhom-data/userdata/media/books` — the directory the capture set looks in. The
|
||
capture log went from `0 mandatory path(s)` to **`1 mandatory path(s)`**.
|
||
- **`ok` means it.** A run that cannot capture a MANDATORY directory now reports **`incomplete`**,
|
||
names the app and the folders, and raises the operator digest — instead of `ok` with a warning
|
||
beside it.
|
||
|
||
**And the sentinel is in the snapshot, listed by name:**
|
||
|
||
```
|
||
$ restic ls -l latest --tag calibre-web
|
||
-rw-r--r-- 1000 1000 181 2026-08-04 12:53:06 /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
|
||
```
|
||
|
||
sha256 **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** — byte-identical to §1,
|
||
verified after the fix's migration moved the file to the corrected directory.
|
||
|
||
**One live check could NOT be reproduced, and is recorded rather than claimed.** Hiding the books
|
||
directory to watch the `incomplete` verdict fire on hardware did not work: the running container's
|
||
bind mount **recreated** the directory, so `os.Stat` succeeded and there was no gap. That is itself
|
||
worth knowing — a bind-mounted directory cannot easily be "missing" while its app runs, so the
|
||
mandatory-gap condition arises in practice when the path resolves somewhere the app never binds (the
|
||
R-203 case), not when a live app's own directory vanishes. The verdict is proven by the run-level
|
||
test, which drives the real `RunOffboxBackup` and asserts both `incomplete` and the operator signal.
|
||
The fixture was restored and the sentinel re-verified at the same hash.
|
||
|
||
## 8. To resume the drill
|
||
|
||
1. **Fix or scope R-203.** Until the mandatory userdata path lands in the snapshot, no sentinel can
|
||
survive a wipe and the drill cannot reach its verdict.
|
||
2. Re-run steps 4–5 and confirm the sentinel IS in the snapshot — by listing it, not by a green status.
|
||
3. Then P4 (the deliberate archive), the §7 STOP, and steps 6–11 as written.
|
||
|
||
Everything else is already in place: the code, the versions, the recovery code, the working repository,
|
||
the file-leg app and the sentinel.
|