admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

REPORT: R-203 shipped and proven live; R-201 unblocked
gates / gates (push) Successful in 8s
Details

Browse Source
This commit is contained in:
authentik Default Admin
2026-08-04 18:53:13 +02:00
parent 73fb595e38
commit 24acc8d816
1 changed files with 149 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+149 -145
View File
@@ -1,168 +1,172 @@
# REPORT — R-201 drill: PREPARED, HALTED BEFORE THE WIPE (2026-08-04) # REPORT — R-203: the app and its backup look in the same place, and "ok" means it (2026-08-04)
**Nothing was wiped. Nothing irreversible happened.** The drill stopped at step 4 because the sentinel **Controller v0.196.0 → v0.197.0**, deployed to demo-hp. No hub change. Nothing deleted, wiped or
file was not in the off-site snapshot — wiping would have destroyed the only copy and proven nothing. moved on any box; the drill was **not** resumed. `demo-felhom` untouched.
--- ---
## 1. THE VERDICT — not reached, and that is the correct outcome ## 1. Confirmed baselines, and §3's landmarks
The pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established the sentinel `felhom-controller` `532f5712a891` (v0.196.0) → **v0.197.0**; `felhom.eu` `a0c4b607a6cf`, no bump.
is not in the off-site snapshot at all**, so there was nothing to recover and the verdict is Both trees clean on arrival. **Every §3 landmark held**, including the one that decided the fix:
unobtainable. The runbook's own rule applies: *a precondition was wrong, and the finding outranks the `paths.go:26` names the system-drive arrangement *"the SSD-only system-data fallback"* — **it is
drill.* supported**, so the resolution was what was wrong and no API refusal was added.
**Sentinel (step 3), still on the box:** ## 2. The call sites found — **FIVE, not four**
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt`, 181 B.
## 2. Snapshot count at step 9 — N/A; at step 4 it was 3, and the sentinel was in none of them | # | Site | Named by the spec? |
```
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
snapshot_count 3 · repo_size_bytes 30 636
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
/mnt/sys_drive/felhom-data/userdata/media/books
[INFO] [offbox] backed up calibre-web (…, 0 mandatory path(s))
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
```
**The run reported `ok`.** → **R-203**, §9.
## 3. Preconditions P1–P6
| # | Result |
|---|---|
| **P1** recovery code held | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file. The code appears nowhere in this report and **was never used** — no unseal ran on demo-hp. |
| **P2** versions | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed published v0.125.0 (sha `f7d8339b53d9…`, verified against the release output) + controller v0.196.0. `age` present at `/usr/bin/age`. |
| **P3** the blob seals the repo under test | **PASS with the caveat that reshaped the drill** — `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`; but **no repository existed under that key** (§5). |
| **P4** verified rollback archive | **NOT TAKEN, deliberately** — only needed for the wipe, and the wipe did not happen. |
| **P5** demo-felhom untouched | **PASS** — not touched at any point. |
| **P6** space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 %, guest 64 GB free. |
## 4. Every step's observable
| step | observable |
|---|---|
| 1 | `identity_blob` 572 B; `restic_pw_sha256` `8a9e33aa4da6…`; `snapshot_count: 0`, **no `last_run`/`last_status` at all** — never ran an off-site backup in this lifetime |
| 1b | **repository ORPHANED** — see §5 |
| 1c | **reset** (operator-authorised) — move-aside to `/home/felhom-repo.orphaned-20260804`, fresh repo initialised, `offbox_repo_reset` pushed |
| 2 | recovery codes for both boxes created and saved by the operator earlier the same day; escrow rows stamped `11:11:37Z` / `11:13:06Z` with **`restic_pw_sha256` unchanged** on both — correctly treated as same-password re-ceremonies (**no superseded row, no `offsite_repo_key_changed`**): v0.93.0's Scenario E, live |
| 3 | `calibre-web` deployed through the real API (HTTP 202), toggled off-site, Tier-1 recovery unit captured; sentinel written + `sync`ed + hashed |
| 4 | `ok`, 3 snapshots, 30 636 B — **and the mandatory directory missing** |
| 5–11 | **NOT RUN.** The §7 STOP was never reached: the drill failed its own precondition first |
## 5. Step 1b — the first live proof of a prediction, and it is the good news
The 2026-08-04 spike predicted from source, and could not measure, that a rebuilt box's next off-site
run would hit a **third** outcome — neither reattaching the old snapshots nor silently starting fresh,
but **refusing**. Measured two ways:
**Read-only probe** (`restic cat config` with the current key, writes nothing):
`Fatal: wrong password or no key found` — the exact string `classifyResticProbe` maps to `orphaned`.
**Then the real customer path** (`POST /backup/offbox/run`):
```
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
no-longer-available key; runs will skip until reset
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
[INFO] Event pushed: offbox_repo_orphaned (warning) …
```
`repo_state: orphaned`, orphan card rendered, event at the hub (HTTP 200).
**The system stopped and said so. It did not quietly start a new history.** Closes R-193's Q3.
**Why it was orphaned:** the 15 snapshots / 40.9 MB were written under key `8e03eddf9ff7…` before the
2026-08-03 rebuild. That key survives only in superseded escrow row **id 3**, whose `identity_blob` is
**NULL** — superseded at `2026-08-04 07:15:36`, **four hours before hub v0.93.0 fixed the retention**.
Permanently unrecoverable, with or without a recovery code.
## 6. Step 7's assertion (`identity_blob` unchanged across the rebuild) — NOT REACHED
There was no rebuild. Unmeasured, and stated as such.
## 7. Did the built chain carry steps 8–9? — NOT REACHED
Neither the check nor the install ran against a live recovery on demo-hp. The manual substitute was not
needed and was not used. **Part 0 remains unit-proven only.**
## 8. Part 0 — files, commit, tests, deployment
**Commit `1b1366b`**, controller **v0.196.0**, deployed to demo-hp guest 9201 (`Up (healthy)`).
`internal/backup/offbox_recovery_cli.go` (`RecoverAndInstall`), `cmd/controller/main.go`
(`--recover-offsite-install` + `--confirm-install`), `internal/backup/offbox_recovery_check_test.go`.
`go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK. Test run and commit were
never combined.
**Red-proof observed:** removing the confirmation gate makes the dry run write the password —
`TestRecoverAndInstall_InstallsOnABareBox` fails with *"the DRY RUN wrote the password"*.
## 9. R-203 — the finding that halted the drill
| what | path | exists? |
|---|---|---| |---|---|---|
| the app's live bind (where the sentinel is) | `/mnt/sys_drive/userdata/media/books` | **YES** | | 1 | `stacks/deploy.go` `withPathVars` → `${USERDATA_PATH}` | yes — the live defect |
| what the off-site capture set looked for | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** | | 2 | `appexport/fabplan.go` | yes |
| 3 | `appexport/export.go` | yes |
| 4 | `stacks/delete.go` `ExportDataMounts` | yes |
| 5 | **`web/handlers.go` FileBrowser mount builder** | **NO** |
**Mechanism, from source:** `NamespaceRoot` (`appbackup/paths.go:28-33`) appends `felhom-data` **when The fifth is the customer's own file browser: on a non-enrolled path it would have mounted the wrong
the drive IS the system data path** (`backup/backup.go:331`); the deploy-time `${USERDATA_PATH}` is directory. **Latent, not live** — the system drive is deliberately never a registered `StoragePath`, so
`<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an the resolver is the identity there today. Wired anyway, with that reason in the code.
app at `HDD_PATH=/mnt/sys_drive`, they differ. **The same compose used both roots** —
`${IMPORT_PATH}` → `…/felhom-data/userdata/import/calibre`, `${USERDATA_PATH}` →
`/mnt/sys_drive/userdata/media/books`.
**Measured vs not, because it changes the fix.** MEASURED: the disagreement, the absence from the Two further sites of the same class were found and fixed: `ComputeFabBuckets` was receiving the drive
snapshot, the `ok` status, and that the only signal is a container-log WARN. **NOT ESTABLISHED:** path where `ComputeCaptureSet` has always received the namespace root, so the export's classified
whether `HDD_PATH=/mnt/sys_drive` is supported — it was chosen because demo-hp's only registered drive paths and the backup's capture set could describe different directories for the same declared bind.
(`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working), while
`/mnt/sys_drive` was **accepted** (202). **Either branch is a defect** and the row says so.
**Not a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths, so **The rule had TWO existing copies and they differed.** `backup.Manager.namespaceRoot` compared
they are unaffected and their snapshots are real. without `filepath.Clean`; `stacks.Manager.inGuest` compared with it. A trailing slash from config
would have flipped the mode in one package and not the other. Both now delegate to
`appbackup.NamespaceRootFor`.
## 10. Teardown — three layers, nothing torn down that proves anything ## 3. Scenario A — the two paths, before and after (live, demo-hp)
| layer | state | ```
before: /mnt/sys_drive/userdata/media/books ← app bind; capture set looked elsewhere
after: /mnt/sys_drive/felhom-data/userdata/media/books ← app bind == capture root
```
Capture log: **`0 mandatory path(s)`** → **`1 mandatory path(s)`**.
## 4. Scenario F — the sentinel in the snapshot's file listing
```
$ restic ls -l latest --tag calibre-web
-rw-r--r-- 1000 1000 181 2026-08-04 12:53:06
/mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
```
And the snapshot's own `paths`:
`["…/backups/primary/calibre-web", "/mnt/sys_drive/felhom-data/userdata/media/books"]`.
**Not a green status — the file, by name and size.** sha256
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`, byte-identical to the drill's
step 3 after the fix's migration moved it into the corrected directory.
## 5. Scenario E — the delete path
**The §8.3 risk does not exist, and this is the correction owed.** `ExportDataMounts` lives in
`delete.go` but is **export-only**: its single production caller is the `.fab` export adapter and
nothing deletes on its result. The delete path's own guard, `ProtectedHDDPaths`, is **layout-agnostic
by construction** — it protects both `<hdd>/…` and `<hdd>/felhom-data/…` — so deletion was never
affected by the namespace-root defect. That note is now in the function's doc comment, because the
file placement misled this change's own specification.
**Does the corrected path point anywhere the old code did not?** Yes, for the export only, and only on
the system-data fallback: `<hdd>/felhom-data/userdata` instead of `<hdd>/userdata`. That is the
directory the app actually binds after this release, and the tests assert the **negative** — no
emitted path lies outside the app's own data roots, on either drive kind. It shipped as its own commit.
## 6. The verdict value
**`incomplete`** — **minted**, because `ok` | `error` | `running` contained nothing meaning *"it ran,
and this app is not fully protected"*. **Not `error`:** the rest of the run worked and what was
captured is real, so `SnapshotCount` and the `LastSuccess` anchor still record it.
It reaches the operator through the **existing** per-run digest, `backup_run_failures` — already
operator-only, already allowlisted. A new event type would have been a two-repo change and the hub
drops anything outside `allowedEventTypes`; the prompt ruled out a hub change. The Hungarian customer
warning is unchanged, and the backups page renders `! Hiányos` with the warn styling.
## 7. §8.4's narrowing — **no customer-visible warning disappears**
`TierOffsite`'s `tierKeeps()` already admits `ClassMandatory` only, so an optional path cannot reach
the stat-filter. The added class check is a **no-op today**, written for parity with Tier 2 — and
**demonstrated to be load-bearing anyway**: widening the tier filter alone keeps the tests green
*because of the check*; widening it and removing the check makes an optional gap start reporting.
## 8. §8.6's live effect — anticipated, and then NOT reproducible
Anticipated in the CHANGELOG before it could fire: calibre-web on demo-hp had exactly this gap, so its
status would become `incomplete`. **In the event it did not**, because the same session fixed the
underlying path — after the corrected bind and the data migration the app has no gap, and the run is
legitimately `ok`.
**Attempting to observe the verdict live by hiding the directory did not work, and that is recorded
rather than dressed up:** the running container's bind mount **recreated** it, so `os.Stat` succeeded
and no gap existed. Worth knowing in itself — a bind-mounted directory cannot easily be "missing"
while its app runs, so the mandatory-gap condition arises in practice when the path resolves somewhere
the app never binds (the R-203 case), not when a live app's own directory vanishes. The verdict is
proven by a run-level test that drives the real `RunOffboxBackup`. The fixture was restored and the
sentinel re-verified at the same hash.
## 9. Tests and every red-proof
| Scenario | Result | Red-proof — mutation → outcome |
|---|---|---|
| A both paths agree | PASS + **LIVE** | restore the bare-path call → **FAIL**: `/mnt/sys_drive/userdata` vs `/mnt/sys_drive/felhom-data/userdata` |
| B enrolled drive unchanged | PASS | invert the drive-kind comparison → **FAIL** on every enrolled row |
| C mandatory gap → not ok | PASS | unreachable gap recording → **FAIL**; unconditional `ok` → **FAIL** |
| D optional gap → ok | PASS | class check removed **with the tier filter widened** → **FAIL**; tier widened alone → PASS, i.e. the check holds the line |
| E export mounts, both kinds + the negative | PASS | leave the site bare → **FAIL**, emits the short path |
| F sentinel in the listing | **LIVE** | — |
**A RED-PROOF PASSED AND THE TEST WAS WRONG, NOT THE CODE** (§9.11 — three of the last six sessions).
My first Scenario-C test exercised `offboxCaptureSet` alone, while the mutation lives in
`runOffboxInternal`. A mutation the test cannot observe is not a red-proof. Replaced with a run-level
test that drives `RunOffboxBackup` and asserts `incomplete`, the retained `LastSuccess`, and the
operator signal; it fails under both mutations. The original Scenario-D "red-proof" also could not
fail by construction — recorded above with the two-part mutation that does.
Green gate after each phase (`go build && go vet && go test ./...`, rc=0) plus
`controller_gates.py --fast`. No test run was combined with a commit.
## 10. Commits
| Commit | Contents |
|---|---| |---|---|
| the guest | `calibre-web` deployed, running, off-site-toggled, sentinel in place. **Kept** — it is the only app on either demo box with a restorable file leg, and the resumed drill needs it | | `73efb09` | Part 1 — one resolver, the four non-export sites, `ComputeFabBuckets`, the two delegating copies |
| the off-site store | fresh, working, 3 snapshots, 30 636 B. Old 15-snapshot history **set aside, not deleted**, at `/home/felhom-repo.orphaned-20260804` (≈41 MB) | | `a96c3d9` | **Part 1.3 alone** — the `delete.go`-resident export-mount site, with its scope correction |
| the host | agent v0.125.0, controller v0.196.0; `pvesm` unchanged beyond normal usage | | `58c703b` | Part 2 — the verdict, the structural gaps, the operator digest, the status rendering |
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` recorded for `demo-hp`. **No scratch customer records were created — nothing was reinstalled** | | `73fb595` | docs (felhom.eu) |
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — ruled, still owed, and §8.3 forbids ## 11. Registers
riding it along with a drill.
## 11. The capability-map row as it now reads - **R-203 → SHIPPED + PROVEN-LIVE.**
- **R-201 → READY TO RESUME**, blocker gone, fixture staged and verified.
- **R-202 stays open**; the orphaned-ciphertext deletion is **still owed**.
- Capability map: new row for mandatory-path capture on both layouts, its predecessor corrected as
optimistic, and the **R-199 back-pointer omitted two sessions ago is now added**.
- The v0.93.0 `identity_blob` retention remains **unit-proven only** — nothing here superseded a key.
The R-199 row (the KEY is recoverable, PROVEN-LIVE) keeps its scope and gains the back-pointer the ## 12. CI
previous update omitted, plus: the attempt to prove the DATA half was **halted before the wipe**, and
**no file has still ever been restored from an off-site backup after a wipe**. The install half
(v0.196.0) is likewise unit-proven only.
## 12. Registers Run numbers and task ids in the session summary. **`--no-verify` was not used.**
- **R-203 NEW** — the missing mandatory directory. Highest ID before this session was R-202. ## 13. Teardown
- **R-201** — prepared, halted, with everything it established and exactly what resuming needs.
- **R-200** — plumbing complete (v0.196.0); the form is not built; the install has never run live.
- **R-202** stays open. **The ciphertext deletion is still owed.**
- **The v0.93.0 `identity_blob` retention is still unit-proven only** — nothing here superseded a key.
## 13. CI **Nothing provisioned.** `calibre-web` and the sentinel stay — R-201 needs them. The temporarily
hidden directory was restored and the sentinel re-verified byte-identical.
Run numbers and task ids in the session summary; **`--no-verify` was not used**.
## 14. Observations — noticed, NOT acted on ## 14. Observations — noticed, NOT acted on
1. **`${IMPORT_PATH}` and `${USERDATA_PATH}` resolved to different roots in one deploy** — the sharpest 1. **`resolveAbs` uses ONE root for two bind classes.** `RootHDD` resolves against the same parameter
single expression of R-203, and the place a fix should probably start. as `RootUserdata`. On an enrolled drive they coincide; on the system-data fallback a `${HDD_PATH}`
2. **The controller's CSRF form field is `_csrf`, not `csrf_token`** — the login page carries one name bind resolves under the namespace root while compose binds it bare. Both callers now pass the
and the protected forms another; an hour went into that. Worth a line in the headless-access memory. namespace root, so the export and the backup **agree with each other** — but whether `${HDD_PATH}`
3. **`/api/stacks/<name>/deploy` accepted `HDD_PATH=/mnt/sys_drive`** one call after refusing the NAS itself should mean the namespace root on the system drive is a separate question that touches every
path with a clear Hungarian error. If the system drive is not a valid app namespace, that refusal is already-deployed app's binds. **Not fixed here; it needs a decision, not a patch.**
missing. 2. **The blast radius was measured before changing anything:** exactly one app in the fleet has
4. **demo-hp has no registered drive usable as an app namespace** — its only drive is a NAS. Any drill `HDD_PATH == system_data_path` — `calibre-web` on demo-hp, deployed for the drill. Every other
needing a file-leg app on that box is constrained by this. deployed app has no `HDD_PATH`. Nothing else needed migrating.
5. **The `felhom-r-leak`-style positive control is now used in three separate absence checks** across 3. **`/api/stacks/<name>/deploy` is first-deployment-only** (409 afterwards), so the corrected
two sessions. It is cheap and it has already caught one wrong test; worth making a documented habit. `${USERDATA_PATH}` reaches an existing app through a start/redeploy, not a re-deploy.
4. **The controller's CSRF form field is `_csrf`, not `csrf_token`** — the login page uses one name and
the protected forms another. Second session in a row this cost time; it belongs in the
headless-access memory.