felhom.eu/documentation/audits/CAMPAIGN-11-recovery-journey-2026-08-05.md

# CAMPAIGN 11 — the recovery journey (2026-08-05 → 08-06)

**Four phases. Phase 1 (the clean journey) FAILED on the journey and PASSED on the data. Phase 3 (key
supersession) PASSED on its central question. Phase 2 (eleven injected faults) and Phase 4 (an
unattended soak) ran overnight on 2026-08-05/06 and are reported here for the first time.**

**Findings: R-214 … R-223 from Phases 1 and 3** (seven since fixed and shipped), **plus R-224 … R-227
from Phase 2.** **Three suspicions investigated — two confirmed, one DISPROVED.** **Four harness
faults, separated from the product's.**

Evidence: `../tests/campaign11-evidence-2026-08-05/` — `journal.md` (Phases 0, 1, 3) and
`journal-phase24.md` (Phases 2 and 4, every observable in the order taken).

> **The one-line answer to the question this campaign was built to ask.** A Hungarian household whose
> machine is rebuilt **gets their data back only if an operator is standing next to them.** The
> cryptography, the retention and the transport all work and are now proven live. **What fails is
> being told the truth**: on this box, four different situations — a mistyped code, a hub outage, a
> stopped agent, and a correct code for a retained earlier package — produce **one** message, and
> three of the four are wrong.

---

## 1. Venue and baselines

| | |
|---|---|
| Host | `demo-hp` (HP t740), **Tier 0**, the designated drill host. Reached **by SSH key, first try** — R-129 stands |
| VM | **321 `c11-appliance`** — q35/OVMF, 4 cores, 8 GB, `cpu=host` |
| Disks | `scsi0` 200 G system · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` |
| Storage | **`c11-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT** (a subdirectory fails the agent's `exactMount` check; Campaign 10 §1) |
| Appliance | `192.168.0.105`, hostname `c11`, `felhom-agent 0.125.0` |
| Guest | LXC **9201**, **`192.168.0.106`** — DHCP, and it MOVED between phases (`.207` → `.227` → `.106`) |
| Hub customer | **`c11`** "Campaign 11", domain `c11.felhom.eu`, DR tier ON, off-site ON |
| Host id | **`c11-36d660`** |
| Untouched | `drill-r50` (VM 300, stopped), guest 9201 on both demo boxes, `local-lvm`, `felhom-backup`, every other hub customer |

### Baselines — every value re-read fresh at the start of Phase 2

| What | Value | How |
|---|---|---|
| `felhom-controller` `main` | **v0.201.0** @ `05cf352a2f17` | `HEAD` == `origin/main`, tree clean |
| `felhom-agent` `main` | **v0.125.0** @ `0404f60e6a7b` | same |
| `felhom.eu` `main` | @ `3a539ea5306a` | same |
| hub, LIVE | **`felhom-hub:0.97.1`** | `kubectl … get deploy hub -o jsonpath` |
| Day-0 manifest | agent **0.125.0** · golden **0.201.0** · `min_agent` **0.125.0** | hub `/configuration`, `selected` options |
| c11 floor | **`v0.200.0 (override)`**; every other customer `v0.156.0` | hub `/configs` |
| Highest register ID | **R-223** | fresh `grep -rhoE 'R-[0-9]{1,3}'` over all four repos |

**All four cited commits match the brief exactly**, and the manifest matches the golden rebake the
Phase-1/3 journal records. **Blast radius of the per-customer floor is still zero, measured** — the
other five customers all read `v0.156.0`.

### The hub CHANGELOG/deployed mismatch — diagnosed more precisely than filed

The brief records *"`0.97.1` shipped without an entry"*. **The content was not missing; its heading
was.** Commit `a7f1d27` wrote the change into the **v0.97.0** entry and `79e31ac` bumped the manifest
to `0.97.1`, so a reader matching the running tag against `hub/CHANGELOG.md` found no `v0.97.1`
heading at all. **Fixed in this session** — the paragraph now has its own entry, marked as added
retroactively. Second occurrence of the class (the first was agent `0.90.1`).

---

## 2. Scope, and what was deliberately not isolated

Campaign 10 could run with Tier 3 OFF; **a campaign about off-site recovery cannot.** Off-site
hard-requires the DR tier, which provisions on **ep0** (Tier 2). The Phase-0 operator ruling stands
and is restated here rather than quietly inherited: **ep0 and the Hetzner Storage Box are written to,
additively** — a PBS namespace and token, a WireGuard peer, and a Storage Box sub-account, all created
on the ordinary customer path. **Nothing existing is modified or deleted.** The brief's I7 wording
("ep0 read-only") was relaxed by that ruling, not widened by this session.

**Phase 2 added no new external writes.** Its faults are network blocks, a service stop, a container
restart and a VM shutdown — all on the campaign's own appliance, all reverted, each with a positive
control proving the fault was real.

---

## 3. The two positives that were owed (brief §4)

### §4.1 — is the floor actually SERVED? **MEASURED. YES. Twice, independently.**

The previous session recorded *"no HELD line and no held reason"* — **two absences** — and correctly
refused to call that a measurement. Both positives were taken here.

**(a) The box's own rendered state.** `/settings` → „Automatikus frissítés – **Minimális verzió
(üzemeltető) `0.200.0`**". That value is `s.updater.GetFloor()`, and `u.floor` has exactly one writer
— `SetFloor`, whose only non-test caller is the report-ACK handler. **Both hold branches of
`ResolveManagedFloor` set `Floor = ""`** (`store.go:2086`, `:2095`), pinned by
`managed_floor_test.go:94`. A non-empty floor on the box therefore proves an ACK carried one.

**(b) A cold-started process saying it out loud.** F8's restart produced the decisive A/B — the same
log line, same box, same code, before and after the golden rebake:

```
15:58:35  settle-gate: GO — floor still unknown after 1m30s               ← hold IN FORCE
21:06:16  settle-gate: GO — at/above floor 0.200.0 (we are 0.201.0)       ← hold RELEASED
```

**The hub half, as a positive rather than an absence:** `managed floor HELD for c11 …` fired every
15 minutes from 17:57:11 to **22:12:06** and then stopped — and the silence is backed by a liveness
control (the hub logged `host-report from demo-hp-bb76ea` at 22:44:41 and `wgsync: pushed 5 peers` at
22:44:57, so it was demonstrably still logging).

> **A correction to the brief's own plan.** It says to take this at DEBUG during F8's restart.
> **A restart alone would not have produced it**: `SetFloor`'s line is `u.dbg(...)`, gated on a private
> flag set from `cfg.Logging.Level == "debug"`, and it writes to the logger — **never to the logx debug
> ring**, so it cannot appear in `/api/debug/logs` at any level. Confirmed live: 4 000 ring entries
> spanning the release window contain no `SetFloor` line, **with a level census run first** (1196
> DEBUG / 2802 INFO / 2 WARN) so the absence was known to be structural rather than evidential.

### §4.2 — R-218's live half: **still NOT measured, deliberately, with the reason**

The fix is present and readable — `needsOffsiteCredential` now retires the declaration on the
**target**, not on the key:

```go
if t != nil { return false }   // a TARGET exists — not a rebuild
```

**The venue cannot exercise it.** c11 has a target (`applied_marker` present since 14:57), so
`t != nil` and the box correctly does **not** declare; declaring here would be the bug. The state that
exercises the fix — hub identity blob present, key placed, **no target** — is shape (a), which the
venue held during Phase 1 and does not hold now. F7's set-aside was examined as a route to it and
**does not produce it either** (§5, F7).

**Recorded as still not measured rather than inferred from the unit test.** It needs one rebuild,
which the brief forbids before Phase 4.

---

## 4. Phase 2 — the eleven faults

**Judged on the message, not the outcome.** Every fault carries a positive control proving the fault
was real; **every control that failed is reported as a failed control, not as a result.**

### The four messages (`internal/web/recovery_handlers.go`, v0.201.0)

| | fires when | first words |
|---|---|---|
| **M1** | unseal failed **and** no superseded package | „A megadott helyreállítási kódot **nem fogadtuk el**. Ellenőrizd, hogy mind a tíz szót…" |
| **M2** | the capability gate says the agent cannot do it (R-216) | „Ez a gép **még nem tudja megnyitni** a mentéseidet… **A kódoddal semmi baj**…" |
| **M3** | unlocked, inventory unreadable (R-217) | „**A kulcs visszakerült**, de a mentések listáját most nem sikerült beolvasni…" |
| **M4** | unseal failed **and** a superseded package is retained (R-222) | „**Ez a kód nem nyitja meg azt a csomagot, amit most őrzünk** ehhez a géphez…" |

**The venue retains a superseded package**, so M4's branch — tested **before** M1 on the failure path
— is live for every failed unlock. That single ordering fact produces two of the four new findings.

### Results

| | Fault | Right answer | Result |
|---|---|---|---|
| **F1** | wrong code ×3 | refused, **M1 only**, no lockout, nothing written | **PARTIAL** — refused ✅, no lockout ✅, nothing written ✅, but **M4, not M1** → **R-226** |
| **F2** | no code at all | states nobody can recover it; offers set-aside | **PASS** |
| **F3** | hub unreachable | names **the hub**, never the code | **FAIL** — M4. → **R-224** |
| **F4** | agent stopped | **M2**, the R-216 fix under pressure | **FAIL** — M4. → **R-224** |
| **F5** | store unreachable after a successful unlock | unlock counts; „could not be read", not „opened with content" | **PASS** — R-217's fix holds |
| **F6** | „Most nem", return later | entry point survives; unlock still works | **PASS** |
| **F7** | set aside, then a change of mind | set aside, never deleted; honest afterwards | *(see §4.7)* |
| **F8** | controller restarted mid-unlock | no half-state; the screen says which | **PARTIAL** — no half-state ✅, but a raw English `Bad Gateway` → **R-227** |
| **F9** | a box that never had off-site | nothing, incl. a direct `GET /recovery` | *(see §4.9)* |
| **F10** | app with a mandatory data path missing | run reports `incomplete`, names the app, digest carries it | *(see §4.10)* |
| **F11** | box offline a whole reporting window | hub and box agree once it returns | *(see §4.11)* |

*(F7, F9, F10 and F11 are written up in the sections below; the table is the index.)*

### The timing discriminator used throughout

`age`'s scrypt makes a real unseal cost ~1 s. **Phase 1's headline was diagnosed by a 0.134 s
response**, and the same instrument separates every fault below:

| | elapsed | what it means |
|---|---|---|
| F1 wrong code | **1.194 / 1.004 / 1.014 s** | a real unseal was attempted and failed |
| F3 hub down | **0.0556 s** | **no unseal attempted** — it failed before the KDF |
| F4 agent down | **0.0299 s** | **no unseal attempted** |
| F5 store down | **1.198 s** | a real unseal, which SUCCEEDED |

**The customer sees the same sentence for the 1.0 s case and the 0.03 s case.**

---

## 5. Findings

### R-224 — every non-code failure on the unlock path is reported as a statement about the code

**F3 and F4 are one defect with two faces**, and it is Phase 1's headline finding relocated from the
version channel to the transport.

| | injected (control) | customer sees | machine's own log |
|---|---|---|---|
| **F3** | hub REJECTed (`302` → `exit 7`) | **M4** | `fetching the sealed bundle: hub: transport error: … no route to host` |
| **F4** | `systemctl stop felhom-agent` (`:8443` gone) | **M4** | `dial tcp 169.254.253.1:8443: connect: connection refused` |

In both the **correct, current** recovery code was entered, so the only possible cause of failure was
the injected fault.

**The discriminator exists and is discarded at the HTTP boundary.** The agent's own `err` separates
the cases exactly —

```
wrong code : "escrow: the recovery code did not unwrap the identity escrow …"
hub down   : "escrow: fetching the sealed bundle: hub: transport error: … no route to host"
```

— but both return **HTTP 400** under one merged sentence (*"the recovery code did not open the sealed
bundle, **or the bundle could not be fetched**"*), the agent's own `msg=` collapses them too, and the
controller's failure path has **no branch for "could not ask / could not reach"** at all. `rerr != nil`
falls straight into the code/package messages.

**Why R-216's gate did not catch F4**, measured rather than reasoned — from the box's debug ring:

```
[web] recovery capability gate: offsite_key_recovery=yes (source=version)
```

**`source=version`.** The gate answers from the *known agent version* without probe traffic, which is
correct for the question it was built for (*is this agent too old?*) and cannot answer the question it
is being used for (*can this machine ask right now?*). **A dead agent of the right version sails
through it** — and the failure that follows is attributed to the code, which R-216's own comment says
must never happen: *"An attempt that cannot succeed must never be made, because its failure is
attributed to the code."*

**Consequence.** During any hub outage or agent restart, a customer holding a perfect recovery code is
told it does not open their package and is routed to customer support about *older* backups. **I6** —
an unreachable service reported as a fact about the code.

### R-225 — the store reports `0 snapshots · 0 GB` when it cannot read it, beside a card saying it holds backups

Found while checking F6's *"the listing is coherent"* clause. `/backups/remote` renders, **on one
screen**:

```
Tároló méret · 0 pillanatkép            Tárhelykeret: 0 / 50 GB (0%)
```

immediately above:

> „**A távoli tároló másik kulccsal készült mentéseket tartalmaz.** … **A meglévő mentések nem
> sérültek**…"

**Ground truth, measured directly against the Storage Box over SFTP** — a read-only listing, no
decryption, using the box's own transport credential:

```
/home/felhom-repo/snapshots:
  f3d9cd67d539c00359f0454ea7a782f45beac8406c55275bee5a6886afa8d791   (Aug 5 13:14)
/home/felhom-repo:   du -s → 12535 KB          /home/felhom-repo/keys:   exactly ONE key
```

That is the Phase 0 snapshot holding **all three sentinels** — the customer's only surviving copy —
matching the journal's `repo_size_bytes 12 611 522`.

**Mechanism, from the box's own state:** after the Phase 3 rebuild the `offbox` block in
`settings.json` carries **no `snapshot_count` and no `repo_size_bytes` key at all**. The values are
*unknown*, and unknown renders as the zero value.

**This is R-217's defect class in a second location** — a field whose zero is indistinguishable from a
real measurement, defaulted past on an unknown path. `OffsiteInventory.Empty` exists precisely because
*"len(Apps)==0 is also what a failed read looks like"*. **I6.**

**I5 checked and NOT breached:** the hub's `/offsite` shows `Campaign 11 · 0.0 GB`, but that is 12.5 MB
rounded to one decimal of a GB and the pool total (`Used 3.8 GB`) is consistent. The two views do not
disagree; **both understate, for different reasons**, and only the box's snapshot **count** — an
integer — is false.

### R-226 — M1, the only message that tells a customer to check their typing, is unreachable on any box that has re-escrowed

The failure path tests M4's condition **before** M1's:

```go
if present, at := s.recoverySuperseded(); present { …M4…; return }
…M1…
```

So on every box the hub keeps an earlier package for, a **genuinely mistyped code** produces M4 —
which is hedged (*„**Ha** egy korábbi kódot adtál meg"*) and states two true facts, but offers no hint
to re-check the ten words and routes the customer to support about older backups.

**Measured:** F1's three wrong-code attempts each returned M4, each after a real ~1 s unseal.

**Why it matters rather than being a nicety:** the population that has re-escrowed is exactly the
population that has just been handed a new recovery code and is most likely to be typing one. R-222's
fix removed one conflation (correct-earlier-code read as a mistype) and introduced another
(mistype read as a correct-earlier-code) **on the same branch**.

### R-227 — a restart mid-unlock returns a raw English `Bad Gateway`

F8 restarted the controller at T+0.7 s, inside the unseal window (control: the container's `StartedAt`
moved). The customer got:

```
HTTP 502 · "Bad Gateway"
```

A raw upstream error, in **English**, from traefik. It names no reason, offers no action, and says
nothing about whether the key was installed. **I3** — *"every refusal names a reason a person can act
on, in Hungarian, with no raw error."* Low severity (the window is ~1 s wide) and recorded rather than
inflated.

---

## 6. What behaved correctly

**A campaign that reports only what broke is half a campaign.** These were tested and held.

| | |
|---|---|
| **R-217's fix** | F5: with the store blocked after a successful unlock, the page rendered M3 and **no listing block at all**. The four false-claim strings („A tároló megnyílt", „van benne tartalom", „nem tudtuk alkalmazásokhoz rendelni") are absent — verified in UTF-8 with two accented positive controls present, after the documented accented-substring false-zero trap was accounted for |
| **No lockout** | F1: three wrong codes, three identical responses, no rate limit, no refusal to try again — as `§8.4 NO LOCKOUT` documents |
| **Nothing written on failure** | F1: all four `/data/offbox` files byte-identical, **mtimes frozen at 14:57:35** |
| **I4** | the entered code appears in **no** log, file or page. Sweep run with a planted-canary control first; the only hits were the harness's own script |
| **The „Most nem" asymmetry** | F6: the full page stops interrupting `/launcher` and `/dashboard`, while `/recovery` stays 200 and the backups-area entry point survives — bound to the offer, never to the postpone flag |
| **No half-state** | F8: a restart mid-unlock left the key files and `settings.json` coherent; the controller returned healthy in 40 s |
| **R-215's fix** | the `GET /recovery` gate is present in v0.201.0 and consults the same predicate as the POST sibling |
| **R-198's retention, in the UI** | the hub host page reads „Key Escrow: present · **1 superseded escrow blob(s) retained**" |
| **The empty submission** | F2: „Add meg a helyreállítási kódot." in 0.027 s — no unseal, and no blame attached to a code never given |

---

## 7. Harness faults, separated from the product's

**Four, all mine, none a product defect** — recorded because Campaign 10's §4d is the format and
because two of them nearly produced false findings.

1. **`source ~/.config/credentials` echoes secrets.** The file holds keys with hyphens
   (`R_DEMO-HP=…`) that bash cannot assign, and the `command not found` error **prints the value**.
   Two demo-box recovery codes were printed this way before the helper was rewritten to `grep` the one
   key it needs. **A live I4 hazard for any session that sources that file** — worth a memory, not a
   register row.
2. **Quoted values in that file** — a bare `cut -d=` keeps the quotes and yields the wrong secret; the
   hub returned `302` until they were stripped. Already in project memory; re-confirmed.
3. **The recovery page carries no `<meta name="csrf-token">`** — it renders a hidden `_csrf` input
   (`data["CSRFField"]`). The first F1 run produced three `403`s and `CSRF-LEN-0`; **the harness's own
   length check caught it**, and the controller's log named the reason word (`token mismatch`).
   Harness, not product.
4. **Two failed reachability controls, reported as failures.** The F5 probe first used `nc`, which the
   controller container does not have, so blocked and unblocked printed the same fallback; the second
   attempt used the IPv6 address `getent hosts` prefers, and **the container has no IPv6 route at
   all**. Only a `/dev/tcp` probe against the IPv4 address (`91.98.242.176`) discriminated
   `TCP-OPEN` → `TCP-CLOSED`. **Two readings that looked like results and were instrument failures.**

---

## 8. Suspicions investigated

| | verdict |
|---|---|
| *"The floor is still held — there is no HELD line and no held reason"* | **DISPROVED.** Both absences were real, and both are explained: the hub stops logging when it stops holding, and `SetFloor`'s DEBUG line cannot reach the debug ring at all. The floor is served, measured two ways (§3) |
| *"The hub and the box disagree about the store's contents (I5)"* | **DISPROVED.** Both understate; the hub's `0.0 GB` is rounding of 12.5 MB. Only the box's snapshot **count** is false → R-225, which is an I6 finding, not an I5 one |
| *"`GET /recovery` still renders on a box it should not"* | **CONFIRMED FIXED** — the gate is present in v0.201.0 (§9, F9) |

---

*(Sections 9–12 — F7, F9, F10, F11, Phase 4, invariants, teardown and hygiene — follow below.)*