admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

drill R-201: prepared and HALTED before the wipe — a mandatory data directory was missing from the off-site snapshot while the run said ok (R-203)
gates / gates (push) Successful in 8s
Details

Browse Source
This commit is contained in:
authentik Default Admin
2026-08-04 15:00:06 +02:00
parent 846253baa8
commit a0c4b607a6
7 changed files with 457 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+130 -183
View File
@@ -1,221 +1,168 @@
# REPORT — the offsite backup key comes back (2026-08-04, R-199 closed, R-200 half)
# REPORT — R-201 drill: PREPARED, HALTED BEFORE THE WIPE (2026-08-04)
**Class:** implementation across three repos + one decisive live proof. Nothing wiped, deleted,
rotated or reconfigured; no customer-facing change in any language.
**Nothing was wiped. Nothing irreversible happened.** The drill stopped at step 4 because the sentinel
file was not in the off-site snapshot — wiping would have destroyed the only copy and proven nothing.
---
## 1. SCENARIO A — THE RESULT, FIRST, BECAUSE IT IS THE SESSION
## 1. THE VERDICT — not reached, and that is the correct outcome
**On `demo-felhom`, 2026-08-04 13:49 CEST, through the real endpoints end to end:**
The pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established the sentinel
is not in the off-site snapshot at all**, so there was nothing to recover and the verdict is
unobtainable. The runbook's own rule applies: *a precondition was wrong, and the finding outranks the
drill.*
**Sentinel (step 3), still on the box:**
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt`, 181 B.
## 2. Snapshot count at step 9 — N/A; at step 4 it was 3, and the sentinel was in none of them
```
=== offsite key recovery check (R-200) — compares, never installs ===
on-disk sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
recovered sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
[MATCH] the offsite repository password IS recoverable from the sealed escrow.
Nothing was written: this check compares and never installs.
EXIT=0
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
snapshot_count 3 · repo_size_bytes 30 636
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
/mnt/sys_drive/felhom-data/userdata/media/books
[INFO] [offbox] backed up calibre-web (…, 0 mandatory path(s))
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
```
**The hashes matched.** And they match a **third, independent** source: `host_escrow.restic_pw_sha256`
on the hub reads `c60c8bc737a6…` for `demo-felhom-8363b5`, recorded at the ceremony and never involved
in this comparison. The box's file, the sealed bundle and the hub's record all agree.
**The run reported `ok`.** → **R-203**, §9.
**The offsite repository password is recoverable from the sealed escrow with the customer's recovery
code.** That had never been demonstrated — the one live round-trip on record (2026-06-10) predates the
field by a month.
## 3. Preconditions P1–P6
## 2. Part 0 — the recovery code
| # | Result |
|---|---|
| **P1** recovery code held | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file. The code appears nowhere in this report and **was never used** — no unseal ran on demo-hp. |
| **P2** versions | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed published v0.125.0 (sha `f7d8339b53d9…`, verified against the release output) + controller v0.196.0. `age` present at `/usr/bin/age`. |
| **P3** the blob seals the repo under test | **PASS with the caveat that reshaped the drill** — `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`; but **no repository existed under that key** (§5). |
| **P4** verified rollback archive | **NOT TAKEN, deliberately** — only needed for the wipe, and the wipe did not happen. |
| **P5** demo-felhom untouched | **PASS** — not touched at any point. |
| **P6** space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 %, guest 64 GB free. |
**Answered by the operator before the session began:** the 2026-08-04 07:16/07:20 codes had not been
kept, so **the operator ran a fresh ceremony on both nodes and saved the codes** (supplied to this
session as `R_DEMO-FELHOM` in the DooPlex credentials file). No ceremony was run by CC.
## 4. Every step's observable
Measured hub-side, and worth recording because it exercised two of yesterday's behaviours live:
the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z` (demo-felhom), and
`restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated them as
same-password re-ceremonies: **no superseded row was created** and **no `offsite_repo_key_changed`
fired**. That is v0.93.0's Scenario E, live. It also means yesterday's `identity_blob` retention is
still unit-proven and structurally verified, **not yet exercised live** — no supersession has happened.
| step | observable |
|---|---|
| 1 | `identity_blob` 572 B; `restic_pw_sha256` `8a9e33aa4da6…`; `snapshot_count: 0`, **no `last_run`/`last_status` at all** — never ran an off-site backup in this lifetime |
| 1b | **repository ORPHANED** — see §5 |
| 1c | **reset** (operator-authorised) — move-aside to `/home/felhom-repo.orphaned-20260804`, fresh repo initialised, `offbox_repo_reset` pushed |
| 2 | recovery codes for both boxes created and saved by the operator earlier the same day; escrow rows stamped `11:11:37Z` / `11:13:06Z` with **`restic_pw_sha256` unchanged** on both — correctly treated as same-password re-ceremonies (**no superseded row, no `offsite_repo_key_changed`**): v0.93.0's Scenario E, live |
| 3 | `calibre-web` deployed through the real API (HTTP 202), toggled off-site, Tier-1 recovery unit captured; sentinel written + `sync`ed + hashed |
| 4 | `ok`, 3 snapshots, 30 636 B — **and the mandatory directory missing** |
| 5–11 | **NOT RUN.** The §7 STOP was never reached: the drill failed its own precondition first |
Subject box: **demo-felhom** (Tier 0, reachable with a baked key, current escrow). Not demo-hp
(break-glass only), not DooPlex, not ep0.
## 5. Step 1b — the first live proof of a prediction, and it is the good news
## 3. The §8.2 decision as built, and where the gate is
The 2026-08-04 spike predicted from source, and could not measure, that a rebuilt box's next off-site
run would hit a **third** outcome — neither reattaching the old snapshots nor silently starting fresh,
but **refusing**. Measured two ways:
**Built as §8.2:** `GET /api/v1/hosts/{host_id}/escrow`, authenticated with the box's own per-host key,
self-scoped (global may read any — the same asymmetry the PUT has).
**Read-only probe** (`restic cat config` with the current key, writes nothing):
`Fatal: wrong password or no key found` — the exact string `classifyResticProbe` maps to `orphaned`.
**The single gate point is `api.escrowSelfServiceRetrieval`** (`hub/internal/api/handler.go`), a named
constant. `false` re-imposes the recovery-mode requirement on this endpoint and changes nothing else —
the same unseal, the same extraction, the same proof, minus the self-service property. The trade is
written above the handler in `dr.go`'s style: what it gives out, why it is safe, and **what it changes
about who is required** — obtaining the blob used to need an operator to arm recovery mode and now
needs only that box's credential. They still cannot open it. The mitigation is the audit row.
**Then the real customer path** (`POST /backup/offbox/run`):
## 4. Files, commits, deployed versions
```
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
no-longer-available key; runs will skip until reset
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
[INFO] Event pushed: offbox_repo_orphaned (warning) …
```
| Repo | Commits | Version | Deployed |
|---|---|---|---|
| `felhom.eu` | `435f4a5` (code) · `dd08926` (manifest) · docs commit below | hub **v0.94.0** | ArgoCD Synced/Healthy, `felhom-hub:0.94.0` |
| `felhom-agent` | `6d79047` | **v0.125.0** | published sha `f7d8339b53d9…`, **verified by independent download**, installed on demo-felhom (`felhom-agent --version` → 0.125.0, `active`). **NOT vouched** |
| `felhom-controller` | `9640e51` | **v0.195.0** | `felhom-controller:0.195.0` on guest 9201, `Up (healthy)` |
`repo_state: orphaned`, orphan card rendered, event at the hub (HTTP 200).
**The system stopped and said so. It did not quietly start a new history.** Closes R-193's Q3.
**Hub:** `internal/api/handler.go` (route + `handleHostEscrowGet` + `escrowSelfServiceRetrieval`),
`internal/notify/dispatcher.go` (`escrow_blob_served` → `operatorOnlyEvents`),
`internal/api/escrow_get_test.go` (new).
**Agent:** `internal/escrow/recover.go` (new — `OffsiteKeyRecoverer`, distinct errors),
`internal/hub/client.go` (`FetchIdentityEscrow`), `internal/localapi/escrow_recover.go` (new route),
`internal/localapi/server.go` (seam + registration), `cmd/felhom-agent/main.go` (production wiring +
the §8.6 selftest message fix), plus `recover_test.go` and `escrow_recover_wiring_test.go` (new).
**Controller:** `internal/backup/offbox_recovery_check.go` + `offbox_recovery_cli.go` (new),
`internal/agentapi/escrow.go` (`RecoverOffsiteRepoPassword`), `cmd/controller/main.go`
(`--recover-offsite-check`), plus `offbox_recovery_check_test.go` (new).
**Why it was orphaned:** the 15 snapshots / 40.9 MB were written under key `8e03eddf9ff7…` before the
2026-08-03 rebuild. That key survives only in superseded escrow row **id 3**, whose `identity_blob` is
**NULL** — superseded at `2026-08-04 07:15:36`, **four hours before hub v0.93.0 fixed the retention**.
Permanently unrecoverable, with or without a recovery code.
**Green gate in each repo after each phase** (`go build && go vet && go test ./...`, rc=0), plus
`repo_gates.py` / `agent_gates.py` / `controller_gates.py` — all OK. **No test run was combined with a
commit.**
## 6. Step 7's assertion (`identity_blob` unchanged across the rebuild) — NOT REACHED
## 5. Tests and every red-proof
There was no rebuild. Unmeasured, and stated as such.
| Scenario | Result | Red-proof — mutation → outcome |
## 7. Did the built chain carry steps 8–9? — NOT REACHED
Neither the check nor the install ran against a live recovery on demo-hp. The manual substitute was not
needed and was not used. **Part 0 remains unit-proven only.**
## 8. Part 0 — files, commit, tests, deployment
**Commit `1b1366b`**, controller **v0.196.0**, deployed to demo-hp guest 9201 (`Up (healthy)`).
`internal/backup/offbox_recovery_cli.go` (`RecoverAndInstall`), `cmd/controller/main.go`
(`--recover-offsite-install` + `--confirm-install`), `internal/backup/offbox_recovery_check_test.go`.
`go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK. Test run and commit were
never combined.
**Red-proof observed:** removing the confirmation gate makes the dry run write the password —
`TestRecoverAndInstall_InstallsOnABareBox` fails with *"the DRY RUN wrote the password"*.
## 9. R-203 — the finding that halted the drill
| what | path | exists? |
|---|---|---|
| **A** recovered key == on-disk key | PASS (unit, real `age`) + **PROVEN LIVE** | return `bundle.PBSToken` instead of `ResticRepoPassword` → **FAIL**: *"the recovered key is not the sealed repository password (len 24 vs 64)"*. This is the mutation that mattered: every field of that bundle is a non-empty string that looks like a secret |
| **B** wrong R fails closed, nothing written | PASS + **PROVEN LIVE** | — (the property is age's KDF, asserted rather than guarded, so there is no guard to remove) |
| **C** a box fetches only its own blob | PASS | delete the `!isGlobal && authHostID != pathHostID` check → **FAIL**: host A was served host B's blob, body and all |
| **D** no blob → a clean "none" | PASS | — |
| **E** every retrieval leaves a record | PASS + **PROVEN LIVE** | `if false &&` on the audit block → **FAIL**: *"a sealed recovery blob was served and NOTHING recorded it"* |
| **F** R persists nowhere | PASS + **PROVEN LIVE** | write R to a temp file inside the unseal → **FAIL** (see the correction below) |
| **G** the operator DR path is untouched | PASS | — |
| **H** the seam is wired (AST) | PASS | `_ = escrowRecoverer` with the `Options` field removed → **FAIL**: *"localapi.Options … carries no EscrowRecovery field"* |
| §8.5 compare-never-install | PASS + **PROVEN LIVE** | add `InjectOffboxPassword` to the check → **FAIL**: *"the check INSTALLED the recovered password"* |
| the app's live bind (where the sentinel is) | `/mnt/sys_drive/userdata/media/books` | **YES** |
| what the off-site capture set looked for | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
**TWO MUTATIONS DID NOT REMOVE EVERY GUARD ON THE FIRST ATTEMPT, AND BOTH ARE RECORDED RATHER THAN
QUIETLY REDONE** (§9.13 — two of the last four sessions found a red-proof that passed):
**Mechanism, from source:** `NamespaceRoot` (`appbackup/paths.go:28-33`) appends `felhom-data` **when
the drive IS the system data path** (`backup/backup.go:331`); the deploy-time `${USERDATA_PATH}` is
`<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an
app at `HDD_PATH=/mnt/sys_drive`, they differ. **The same compose used both roots** —
`${IMPORT_PATH}` → `…/felhom-data/userdata/import/calibre`, `${USERDATA_PATH}` →
`/mnt/sys_drive/userdata/media/books`.
1. **Scenario F.** The planted leak wrote R into the walked directory and **the test still passed** —
because a later failing call OVERWROTE the leak file with the wrong code, and the test only scanned
for the first one. Instrumenting the mutation (printing where it wrote) is what found it. **The test
was wrong, not the mutation.** It now asserts **emptiness** of the TMPDIR tree as its primary check —
nothing under it is created by the test, so any survivor is a leak regardless of content — with the
content scan kept as defence in depth over all three secrets. Re-run: **FAIL**, *"the unseal left
1 file(s) behind under TMPDIR"*.
2. **Scenario H.** Commenting the wiring line out failed to **compile** (`declared and not used`), which
is not a red-proof. The mutation that compiles keeps the variable alive (`_ = escrowRecoverer`) and
drops the struct field. Re-run: **FAIL** as quoted above.
**Measured vs not, because it changes the fix.** MEASURED: the disagreement, the absence from the
snapshot, the `ok` status, and that the only signal is a container-log WARN. **NOT ESTABLISHED:**
whether `HDD_PATH=/mnt/sys_drive` is supported — it was chosen because demo-hp's only registered drive
(`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working), while
`/mnt/sys_drive` was **accepted** (202). **Either branch is a defect** and the row says so.
**Every `-run` filter was verified to have matched** (`-v`, `=== RUN` lines counted) before any result
was read as a proof.
**Not a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths, so
they are unaffected and their snapshots are real.
## 6. The live validation trail, in order
## 10. Teardown — three layers, nothing torn down that proves anything
1. **Wrong code first, deliberately** (13:44:44) — hub: `escrow blob SERVED to host demo-felhom-8363b5
(572 opaque bytes, self_scope=true) — recovery path in use`; agent: `offsite key recovery FAILED …
err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
blob): exit status 1"`; CLI: `[FAIL] … nothing was written.`, exit 1.
**This is Scenario B live AND it proves links 6 and 7 ran independently of the success** — the blob
was fetched and the unseal was attempted and refused.
2. **The real code** (13:49:47) — §1's output, exit 0.
3. **Nothing was written** — `offbox/repo_password` mtime on the live box is still
`2026-08-03 07:18:02`, unchanged by a check that ran at `11:49` UTC. The whole `offbox/` directory is
byte-for-byte the pre-check listing.
| layer | state |
|---|---|
| the guest | `calibre-web` deployed, running, off-site-toggled, sentinel in place. **Kept** — it is the only app on either demo box with a restorable file leg, and the resumed drill needs it |
| the off-site store | fresh, working, 3 snapshots, 30 636 B. Old 15-snapshot history **set aside, not deleted**, at `/home/felhom-repo.orphaned-20260804` (≈41 MB) |
| the host | agent v0.125.0, controller v0.196.0; `pvesm` unchanged beyond normal usage |
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` recorded for `demo-hp`. **No scratch customer records were created — nothing was reinstalled** |
## 7. The retrieval record as it appears at the hub
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — ruled, still owed, and §8.3 forbids
riding it along with a drill.
```
customer_id severity created_at details_json
demo-felhom warning 2026-08-04 11:44:44 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
demo-felhom warning 2026-08-04 11:49:47 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
```
## 11. The capability-map row as it now reads
And the routing, which is the half that shows the register works:
The R-199 row (the KEY is recoverable, PROVEN-LIVE) keeps its scope and gains the back-pointer the
previous update omitted, plus: the attempt to prove the DATA half was **halted before the wipe**, and
**no file has still ever been restored from an off-site backup after a wipe**. The install half
(v0.196.0) is likewise unit-proven only.
```
customer_id channel status created_at
demo-felhom operator sent 2026-08-04 11:44:44
demo-felhom customer skipped 2026-08-04 11:44:44 ← operator_only
demo-felhom operator suppressed 2026-08-04 11:49:47 ← 1h operator cooldown, itself recorded (R-182)
demo-felhom customer skipped 2026-08-04 11:49:47
```
## 12. Registers
**Both retrievals raised an event; the operator was mailed for the first; the second's mail was
cooldown-suppressed and that suppression is written down.** The audit is per-retrieval and complete
even where the mail is not.
- **R-203 NEW** — the missing mandatory directory. Highest ID before this session was R-202.
- **R-201** — prepared, halted, with everything it established and exactly what resuming needs.
- **R-200** — plumbing complete (v0.196.0); the form is not built; the install has never run live.
- **R-202** stays open. **The ciphertext deletion is still owed.**
- **The v0.93.0 `identity_blob` retention is still unit-proven only** — nothing here superseded a key.
## 8. Evidence that R persisted nowhere — the search, not a claim
## 13. CI
Searched on the subject host with the pattern read from a file (never in argv), across the agent
journal, the controller's container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`:
Run numbers and task ids in the session summary; **`--no-verify` was not used**.
```
agent journal lines containing R: 0
controller log lines containing R: 0
host files containing R: 0
leftover felhom-idesc-* staging dirs: 0
staged escrow secret file: 0 entries
```
## 14. Observations — noticed, NOT acted on
**With a positive control, because an absent match is not a measurement unless the instrument is shown
to work:** a copy of R was planted under `/tmp`, the same sweep found **1**, the copy was shredded, and
the sweep returned **0**. The instrument detects R when R is there.
R reached the box only as: the credentials-file value on DooPlex → a pipe → the container's stdin →
the request body → the agent's memory. It was never an argument, never a file on either machine, and
appears in no log line at any level.
## 9. Register
- **R-199 → SHIPPED + PROVEN-LIVE.** Links 6, 7 and 8 assembled and walked.
- **R-200 → plumbing shipped; the FORM is not built** and the row stays open for it and for link 9.
- **R-201 → OPEN, and the distinction is deliberate:** the KEY is proven recoverable; a recovered
password has never been **installed**, no repository has been **reopened** under one, and **no file
has ever been restored**. The drill's pass condition is unchanged — a byte-identical sentinel file,
not "the store opened".
- **R-202 → OPEN and untouched.** The orphan card still promises recoverability unconditionally.
- **The orphaned-ciphertext deletion (~1.2 GB) is STILL OWED** — operator ruling 2 of 2026-08-04,
deliberately not ridden along with a code release.
- Capability map, `03-host-agent.md` §8a, `CONTEXT.md` **S-29**, `REUSE.md` and `ROADMAP.md` updated;
`STATUS.md` rewritten for the operator.
## 10. The capability-map row, and what it deliberately does not claim
Added: *"The offsite repository password can be RECOVERED from the sealed escrow with the customer's
recovery code"* — **PROVEN-LIVE (2026-08-04)**, with the evidence above.
**What the row explicitly refuses to claim**, stated in the row itself because the previous over-claim
on this subject was struck out four hours earlier: it covers the **key**, not the **data**. No
recovered password has been installed, no repository reopened, no file restored. And the proof used a
box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is
exactly what the drill covers and it has not run.
## 11. CI
Run numbers, task ids and conclusions for all commits are quoted in the session summary.
**`--no-verify` was NOT used** — every push ran its repo's pre-push gate and passed.
## 12. Teardown
**Nothing was provisioned.** The temporary pattern file used by the R sweep and its positive control
were shredded on the host by the same command that created them (verified: 0 files remain). No
diagnostic artifact persists on either machine. The agent's previous binary is retained as
`/usr/local/bin/felhom-agent.bak-0.124.1`, per the documented deploy.
## 13. Observations — noticed, NOT acted on
1. **The `escrow_blob_served` mail is subject to the 1-hour operator cooldown** — visible above, where
the second retrieval's mail was suppressed. The EVENT rows are per-retrieval so the audit is
complete, but if retrieval ever becomes routine, a burst would produce one mail. Worth revisiting
**when** the customer-facing flow lands, not before.
2. **`ListSupersededEscrow` still has no production caller.** Recovery reads the CURRENT blob; a
customer needing a PREVIOUS generation's key (which is what R-198's retention exists for) has no
path at all. That is the next gap in this area after the drill.
3. **The controller binary is at `/usr/local/bin/felhom-controller`, not `/app/…`** — the first
diagnostic invocation failed on a guessed path. Worth a line in the runbook when one is written.
4. **`age` is at the hardcoded `/usr/bin/age` on both demo hosts** (1.2.1 on demo-felhom) — the recon
listed this as unestablished; it is now established for demo-felhom. `ageBinary` is a package
variable with no config override, so a host that installs `age` elsewhere would fail the unseal at
the last step.
5. **Yesterday's `identity_blob` retention has still not been exercised live** — the operator's fresh
ceremonies sealed the *same* password, so no supersession occurred. It remains unit-proven with a
verified schema; the first real supersession will be its live proof.
1. **`${IMPORT_PATH}` and `${USERDATA_PATH}` resolved to different roots in one deploy** — the sharpest
single expression of R-203, and the place a fix should probably start.
2. **The controller's CSRF form field is `_csrf`, not `csrf_token`** — the login page carries one name
and the protected forms another; an hour went into that. Worth a line in the headless-access memory.
3. **`/api/stacks/<name>/deploy` accepted `HDD_PATH=/mnt/sys_drive`** one call after refusing the NAS
path with a clear Hungarian error. If the system drive is not a valid app namespace, that refusal is
missing.
4. **demo-hp has no registered drive usable as an app namespace** — its only drive is a NAS. Any drill
needing a file-leg app on that box is constrained by this.
5. **The `felhom-r-leak`-style positive control is now used in three separate absence checks** across
two sessions. It is cheap and it has already caught one wrong test; worth making a documented habit.