diff --git a/REPORT.md b/REPORT.md index c0a72a7..4e1278a 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,168 +1,172 @@ -# REPORT — R-201 drill: PREPARED, HALTED BEFORE THE WIPE (2026-08-04) +# REPORT — R-203: the app and its backup look in the same place, and "ok" means it (2026-08-04) -**Nothing was wiped. Nothing irreversible happened.** The drill stopped at step 4 because the sentinel -file was not in the off-site snapshot — wiping would have destroyed the only copy and proven nothing. +**Controller v0.196.0 → v0.197.0**, deployed to demo-hp. No hub change. Nothing deleted, wiped or +moved on any box; the drill was **not** resumed. `demo-felhom` untouched. --- -## 1. THE VERDICT — not reached, and that is the correct outcome +## 1. Confirmed baselines, and §3's landmarks -The pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established the sentinel -is not in the off-site snapshot at all**, so there was nothing to recover and the verdict is -unobtainable. The runbook's own rule applies: *a precondition was wrong, and the finding outranks the -drill.* +`felhom-controller` `532f5712a891` (v0.196.0) → **v0.197.0**; `felhom.eu` `a0c4b607a6cf`, no bump. +Both trees clean on arrival. **Every §3 landmark held**, including the one that decided the fix: +`paths.go:26` names the system-drive arrangement *"the SSD-only system-data fallback"* — **it is +supported**, so the resolution was what was wrong and no API refusal was added. -**Sentinel (step 3), still on the box:** -`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` -at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt`, 181 B. +## 2. The call sites found — **FIVE, not four** -## 2. Snapshot count at step 9 — N/A; at step 4 it was 3, and the sentinel was in none of them - -``` -last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped -snapshot_count 3 · repo_size_bytes 30 636 -[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite: - /mnt/sys_drive/felhom-data/userdata/media/books -[INFO] [offbox] backed up calibre-web (…, 0 mandatory path(s)) -[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s -``` - -**The run reported `ok`.** → **R-203**, §9. - -## 3. Preconditions P1–P6 - -| # | Result | -|---|---| -| **P1** recovery code held | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file. The code appears nowhere in this report and **was never used** — no unseal ran on demo-hp. | -| **P2** versions | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed published v0.125.0 (sha `f7d8339b53d9…`, verified against the release output) + controller v0.196.0. `age` present at `/usr/bin/age`. | -| **P3** the blob seals the repo under test | **PASS with the caveat that reshaped the drill** — `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`; but **no repository existed under that key** (§5). | -| **P4** verified rollback archive | **NOT TAKEN, deliberately** — only needed for the wipe, and the wipe did not happen. | -| **P5** demo-felhom untouched | **PASS** — not touched at any point. | -| **P6** space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 %, guest 64 GB free. | - -## 4. Every step's observable - -| step | observable | -|---|---| -| 1 | `identity_blob` 572 B; `restic_pw_sha256` `8a9e33aa4da6…`; `snapshot_count: 0`, **no `last_run`/`last_status` at all** — never ran an off-site backup in this lifetime | -| 1b | **repository ORPHANED** — see §5 | -| 1c | **reset** (operator-authorised) — move-aside to `/home/felhom-repo.orphaned-20260804`, fresh repo initialised, `offbox_repo_reset` pushed | -| 2 | recovery codes for both boxes created and saved by the operator earlier the same day; escrow rows stamped `11:11:37Z` / `11:13:06Z` with **`restic_pw_sha256` unchanged** on both — correctly treated as same-password re-ceremonies (**no superseded row, no `offsite_repo_key_changed`**): v0.93.0's Scenario E, live | -| 3 | `calibre-web` deployed through the real API (HTTP 202), toggled off-site, Tier-1 recovery unit captured; sentinel written + `sync`ed + hashed | -| 4 | `ok`, 3 snapshots, 30 636 B — **and the mandatory directory missing** | -| 5–11 | **NOT RUN.** The §7 STOP was never reached: the drill failed its own precondition first | - -## 5. Step 1b — the first live proof of a prediction, and it is the good news - -The 2026-08-04 spike predicted from source, and could not measure, that a rebuilt box's next off-site -run would hit a **third** outcome — neither reattaching the old snapshots nor silently starting fresh, -but **refusing**. Measured two ways: - -**Read-only probe** (`restic cat config` with the current key, writes nothing): -`Fatal: wrong password or no key found` — the exact string `classifyResticProbe` maps to `orphaned`. - -**Then the real customer path** (`POST /backup/offbox/run`): - -``` -[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous, - no-longer-available key; runs will skip until reset -[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset) -[INFO] Event pushed: offbox_repo_orphaned (warning) … -``` - -`repo_state: orphaned`, orphan card rendered, event at the hub (HTTP 200). -**The system stopped and said so. It did not quietly start a new history.** Closes R-193's Q3. - -**Why it was orphaned:** the 15 snapshots / 40.9 MB were written under key `8e03eddf9ff7…` before the -2026-08-03 rebuild. That key survives only in superseded escrow row **id 3**, whose `identity_blob` is -**NULL** — superseded at `2026-08-04 07:15:36`, **four hours before hub v0.93.0 fixed the retention**. -Permanently unrecoverable, with or without a recovery code. - -## 6. Step 7's assertion (`identity_blob` unchanged across the rebuild) — NOT REACHED - -There was no rebuild. Unmeasured, and stated as such. - -## 7. Did the built chain carry steps 8–9? — NOT REACHED - -Neither the check nor the install ran against a live recovery on demo-hp. The manual substitute was not -needed and was not used. **Part 0 remains unit-proven only.** - -## 8. Part 0 — files, commit, tests, deployment - -**Commit `1b1366b`**, controller **v0.196.0**, deployed to demo-hp guest 9201 (`Up (healthy)`). -`internal/backup/offbox_recovery_cli.go` (`RecoverAndInstall`), `cmd/controller/main.go` -(`--recover-offsite-install` + `--confirm-install`), `internal/backup/offbox_recovery_check_test.go`. - -`go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK. Test run and commit were -never combined. - -**Red-proof observed:** removing the confirmation gate makes the dry run write the password — -`TestRecoverAndInstall_InstallsOnABareBox` fails with *"the DRY RUN wrote the password"*. - -## 9. R-203 — the finding that halted the drill - -| what | path | exists? | +| # | Site | Named by the spec? | |---|---|---| -| the app's live bind (where the sentinel is) | `/mnt/sys_drive/userdata/media/books` | **YES** | -| what the off-site capture set looked for | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** | +| 1 | `stacks/deploy.go` `withPathVars` → `${USERDATA_PATH}` | yes — the live defect | +| 2 | `appexport/fabplan.go` | yes | +| 3 | `appexport/export.go` | yes | +| 4 | `stacks/delete.go` `ExportDataMounts` | yes | +| 5 | **`web/handlers.go` FileBrowser mount builder** | **NO** | -**Mechanism, from source:** `NamespaceRoot` (`appbackup/paths.go:28-33`) appends `felhom-data` **when -the drive IS the system data path** (`backup/backup.go:331`); the deploy-time `${USERDATA_PATH}` is -`/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an -app at `HDD_PATH=/mnt/sys_drive`, they differ. **The same compose used both roots** — -`${IMPORT_PATH}` → `…/felhom-data/userdata/import/calibre`, `${USERDATA_PATH}` → -`/mnt/sys_drive/userdata/media/books`. +The fifth is the customer's own file browser: on a non-enrolled path it would have mounted the wrong +directory. **Latent, not live** — the system drive is deliberately never a registered `StoragePath`, so +the resolver is the identity there today. Wired anyway, with that reason in the code. -**Measured vs not, because it changes the fix.** MEASURED: the disagreement, the absence from the -snapshot, the `ok` status, and that the only signal is a container-log WARN. **NOT ESTABLISHED:** -whether `HDD_PATH=/mnt/sys_drive` is supported — it was chosen because demo-hp's only registered drive -(`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working), while -`/mnt/sys_drive` was **accepted** (202). **Either branch is a defect** and the row says so. +Two further sites of the same class were found and fixed: `ComputeFabBuckets` was receiving the drive +path where `ComputeCaptureSet` has always received the namespace root, so the export's classified +paths and the backup's capture set could describe different directories for the same declared bind. -**Not a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths, so -they are unaffected and their snapshots are real. +**The rule had TWO existing copies and they differed.** `backup.Manager.namespaceRoot` compared +without `filepath.Clean`; `stacks.Manager.inGuest` compared with it. A trailing slash from config +would have flipped the mode in one package and not the other. Both now delegate to +`appbackup.NamespaceRootFor`. -## 10. Teardown — three layers, nothing torn down that proves anything +## 3. Scenario A — the two paths, before and after (live, demo-hp) -| layer | state | +``` +before: /mnt/sys_drive/userdata/media/books ← app bind; capture set looked elsewhere +after: /mnt/sys_drive/felhom-data/userdata/media/books ← app bind == capture root +``` + +Capture log: **`0 mandatory path(s)`** → **`1 mandatory path(s)`**. + +## 4. Scenario F — the sentinel in the snapshot's file listing + +``` +$ restic ls -l latest --tag calibre-web +-rw-r--r-- 1000 1000 181 2026-08-04 12:53:06 + /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt +``` + +And the snapshot's own `paths`: +`["…/backups/primary/calibre-web", "/mnt/sys_drive/felhom-data/userdata/media/books"]`. + +**Not a green status — the file, by name and size.** sha256 +`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`, byte-identical to the drill's +step 3 after the fix's migration moved it into the corrected directory. + +## 5. Scenario E — the delete path + +**The §8.3 risk does not exist, and this is the correction owed.** `ExportDataMounts` lives in +`delete.go` but is **export-only**: its single production caller is the `.fab` export adapter and +nothing deletes on its result. The delete path's own guard, `ProtectedHDDPaths`, is **layout-agnostic +by construction** — it protects both `/…` and `/felhom-data/…` — so deletion was never +affected by the namespace-root defect. That note is now in the function's doc comment, because the +file placement misled this change's own specification. + +**Does the corrected path point anywhere the old code did not?** Yes, for the export only, and only on +the system-data fallback: `/felhom-data/userdata` instead of `/userdata`. That is the +directory the app actually binds after this release, and the tests assert the **negative** — no +emitted path lies outside the app's own data roots, on either drive kind. It shipped as its own commit. + +## 6. The verdict value + +**`incomplete`** — **minted**, because `ok` | `error` | `running` contained nothing meaning *"it ran, +and this app is not fully protected"*. **Not `error`:** the rest of the run worked and what was +captured is real, so `SnapshotCount` and the `LastSuccess` anchor still record it. + +It reaches the operator through the **existing** per-run digest, `backup_run_failures` — already +operator-only, already allowlisted. A new event type would have been a two-repo change and the hub +drops anything outside `allowedEventTypes`; the prompt ruled out a hub change. The Hungarian customer +warning is unchanged, and the backups page renders `! Hiányos` with the warn styling. + +## 7. §8.4's narrowing — **no customer-visible warning disappears** + +`TierOffsite`'s `tierKeeps()` already admits `ClassMandatory` only, so an optional path cannot reach +the stat-filter. The added class check is a **no-op today**, written for parity with Tier 2 — and +**demonstrated to be load-bearing anyway**: widening the tier filter alone keeps the tests green +*because of the check*; widening it and removing the check makes an optional gap start reporting. + +## 8. §8.6's live effect — anticipated, and then NOT reproducible + +Anticipated in the CHANGELOG before it could fire: calibre-web on demo-hp had exactly this gap, so its +status would become `incomplete`. **In the event it did not**, because the same session fixed the +underlying path — after the corrected bind and the data migration the app has no gap, and the run is +legitimately `ok`. + +**Attempting to observe the verdict live by hiding the directory did not work, and that is recorded +rather than dressed up:** the running container's bind mount **recreated** it, so `os.Stat` succeeded +and no gap existed. Worth knowing in itself — a bind-mounted directory cannot easily be "missing" +while its app runs, so the mandatory-gap condition arises in practice when the path resolves somewhere +the app never binds (the R-203 case), not when a live app's own directory vanishes. The verdict is +proven by a run-level test that drives the real `RunOffboxBackup`. The fixture was restored and the +sentinel re-verified at the same hash. + +## 9. Tests and every red-proof + +| Scenario | Result | Red-proof — mutation → outcome | +|---|---|---| +| A both paths agree | PASS + **LIVE** | restore the bare-path call → **FAIL**: `/mnt/sys_drive/userdata` vs `/mnt/sys_drive/felhom-data/userdata` | +| B enrolled drive unchanged | PASS | invert the drive-kind comparison → **FAIL** on every enrolled row | +| C mandatory gap → not ok | PASS | unreachable gap recording → **FAIL**; unconditional `ok` → **FAIL** | +| D optional gap → ok | PASS | class check removed **with the tier filter widened** → **FAIL**; tier widened alone → PASS, i.e. the check holds the line | +| E export mounts, both kinds + the negative | PASS | leave the site bare → **FAIL**, emits the short path | +| F sentinel in the listing | **LIVE** | — | + +**A RED-PROOF PASSED AND THE TEST WAS WRONG, NOT THE CODE** (§9.11 — three of the last six sessions). +My first Scenario-C test exercised `offboxCaptureSet` alone, while the mutation lives in +`runOffboxInternal`. A mutation the test cannot observe is not a red-proof. Replaced with a run-level +test that drives `RunOffboxBackup` and asserts `incomplete`, the retained `LastSuccess`, and the +operator signal; it fails under both mutations. The original Scenario-D "red-proof" also could not +fail by construction — recorded above with the two-part mutation that does. + +Green gate after each phase (`go build && go vet && go test ./...`, rc=0) plus +`controller_gates.py --fast`. No test run was combined with a commit. + +## 10. Commits + +| Commit | Contents | |---|---| -| the guest | `calibre-web` deployed, running, off-site-toggled, sentinel in place. **Kept** — it is the only app on either demo box with a restorable file leg, and the resumed drill needs it | -| the off-site store | fresh, working, 3 snapshots, 30 636 B. Old 15-snapshot history **set aside, not deleted**, at `/home/felhom-repo.orphaned-20260804` (≈41 MB) | -| the host | agent v0.125.0, controller v0.196.0; `pvesm` unchanged beyond normal usage | -| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` recorded for `demo-hp`. **No scratch customer records were created — nothing was reinstalled** | +| `73efb09` | Part 1 — one resolver, the four non-export sites, `ComputeFabBuckets`, the two delegating copies | +| `a96c3d9` | **Part 1.3 alone** — the `delete.go`-resident export-mount site, with its scope correction | +| `58c703b` | Part 2 — the verdict, the structural gaps, the operator digest, the status rendering | +| `73fb595` | docs (felhom.eu) | -**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — ruled, still owed, and §8.3 forbids -riding it along with a drill. +## 11. Registers -## 11. The capability-map row as it now reads +- **R-203 → SHIPPED + PROVEN-LIVE.** +- **R-201 → READY TO RESUME**, blocker gone, fixture staged and verified. +- **R-202 stays open**; the orphaned-ciphertext deletion is **still owed**. +- Capability map: new row for mandatory-path capture on both layouts, its predecessor corrected as + optimistic, and the **R-199 back-pointer omitted two sessions ago is now added**. +- The v0.93.0 `identity_blob` retention remains **unit-proven only** — nothing here superseded a key. -The R-199 row (the KEY is recoverable, PROVEN-LIVE) keeps its scope and gains the back-pointer the -previous update omitted, plus: the attempt to prove the DATA half was **halted before the wipe**, and -**no file has still ever been restored from an off-site backup after a wipe**. The install half -(v0.196.0) is likewise unit-proven only. +## 12. CI -## 12. Registers +Run numbers and task ids in the session summary. **`--no-verify` was not used.** -- **R-203 NEW** — the missing mandatory directory. Highest ID before this session was R-202. -- **R-201** — prepared, halted, with everything it established and exactly what resuming needs. -- **R-200** — plumbing complete (v0.196.0); the form is not built; the install has never run live. -- **R-202** stays open. **The ciphertext deletion is still owed.** -- **The v0.93.0 `identity_blob` retention is still unit-proven only** — nothing here superseded a key. +## 13. Teardown -## 13. CI - -Run numbers and task ids in the session summary; **`--no-verify` was not used**. +**Nothing provisioned.** `calibre-web` and the sentinel stay — R-201 needs them. The temporarily +hidden directory was restored and the sentinel re-verified byte-identical. ## 14. Observations — noticed, NOT acted on -1. **`${IMPORT_PATH}` and `${USERDATA_PATH}` resolved to different roots in one deploy** — the sharpest - single expression of R-203, and the place a fix should probably start. -2. **The controller's CSRF form field is `_csrf`, not `csrf_token`** — the login page carries one name - and the protected forms another; an hour went into that. Worth a line in the headless-access memory. -3. **`/api/stacks//deploy` accepted `HDD_PATH=/mnt/sys_drive`** one call after refusing the NAS - path with a clear Hungarian error. If the system drive is not a valid app namespace, that refusal is - missing. -4. **demo-hp has no registered drive usable as an app namespace** — its only drive is a NAS. Any drill - needing a file-leg app on that box is constrained by this. -5. **The `felhom-r-leak`-style positive control is now used in three separate absence checks** across - two sessions. It is cheap and it has already caught one wrong test; worth making a documented habit. +1. **`resolveAbs` uses ONE root for two bind classes.** `RootHDD` resolves against the same parameter + as `RootUserdata`. On an enrolled drive they coincide; on the system-data fallback a `${HDD_PATH}` + bind resolves under the namespace root while compose binds it bare. Both callers now pass the + namespace root, so the export and the backup **agree with each other** — but whether `${HDD_PATH}` + itself should mean the namespace root on the system drive is a separate question that touches every + already-deployed app's binds. **Not fixed here; it needs a decision, not a patch.** +2. **The blast radius was measured before changing anything:** exactly one app in the fleet has + `HDD_PATH == system_data_path` — `calibre-web` on demo-hp, deployed for the drill. Every other + deployed app has no `HDD_PATH`. Nothing else needed migrating. +3. **`/api/stacks//deploy` is first-deployment-only** (409 afterwards), so the corrected + `${USERDATA_PATH}` reaches an existing app through a start/redeploy, not a re-deploy. +4. **The controller's CSRF form field is `_csrf`, not `csrf_token`** — the login page uses one name and + the protected forms another. Second session in a row this cost time; it belongs in the + headless-access memory.