felhom.eu/documentation/audits/CAMPAIGN-11-recovery-journey-2026-08-05.md

CAMPAIGN 11 — the recovery journey (2026-08-05 → 08-06)

Four phases. Phase 1 (the clean journey) FAILED on the journey and PASSED on the data. Phase 3 (key supersession) PASSED on its central question. Phase 2 (eleven injected faults) and Phase 4 (an unattended soak) ran overnight on 2026-08-05/06 and are reported here for the first time.

Findings: R-214 … R-223 from Phases 1 and 3 (seven since fixed and shipped), plus R-224 … R-227 from Phase 2. Three suspicions investigated — two confirmed, one DISPROVED. Four harness faults, separated from the product's.

Evidence: ../tests/campaign11-evidence-2026-08-05/ — journal.md (Phases 0, 1, 3) and journal-phase24.md (Phases 2 and 4, every observable in the order taken).

The one-line answer to the question this campaign was built to ask. A Hungarian household whose machine is rebuilt gets their data back only if an operator is standing next to them. The cryptography, the retention and the transport all work and are now proven live. What fails is being told the truth: on this box, four different situations — a mistyped code, a hub outage, a stopped agent, and a correct code for a retained earlier package — produce one message, and three of the four are wrong.

---

1. Venue and baselines

Host	demo-hp (HP t740), Tier 0, the designated drill host. Reached by SSH key, first try — R-129 stands
VM	321 c11-appliance — q35/OVMF, 4 cores, 8 GB, cpu=host
Disks	scsi0 200 G system · scsi1 50 G · scsi2 50 G, qcow2 on c11-scratch
Storage	c11-scratch, dir at /mnt/nvme-1tb — the mount ROOT (a subdirectory fails the agent's exactMount check; Campaign 10 §1)
Appliance	192.168.0.105, hostname c11, felhom-agent 0.125.0
Guest	LXC 9201, 192.168.0.106 — DHCP, and it MOVED between phases (.207 → .227 → .106)
Hub customer	c11 "Campaign 11", domain c11.felhom.eu, DR tier ON, off-site ON
Host id	c11-36d660
Untouched	drill-r50 (VM 300, stopped), guest 9201 on both demo boxes, local-lvm, felhom-backup, every other hub customer

Baselines — every value re-read fresh at the start of Phase 2

What	Value	How
felhom-controller main	v0.201.0 @ 05cf352a2f17	HEAD == origin/main, tree clean
felhom-agent main	v0.125.0 @ 0404f60e6a7b	same
felhom.eu main	@ 3a539ea5306a	same
hub, LIVE	felhom-hub:0.97.1	kubectl … get deploy hub -o jsonpath
Day-0 manifest	agent 0.125.0 · golden 0.201.0 · min_agent 0.125.0	hub /configuration, selected options
c11 floor	v0.200.0 (override); every other customer v0.156.0	hub /configs
Highest register ID	R-223	fresh grep -rhoE 'R-[0-9]{1,3}' over all four repos

All four cited commits match the brief exactly, and the manifest matches the golden rebake the Phase-1/3 journal records. Blast radius of the per-customer floor is still zero, measured — the other five customers all read v0.156.0.

The hub CHANGELOG/deployed mismatch — diagnosed more precisely than filed

The brief records "0.97.1 shipped without an entry". The content was not missing; its heading was. Commit a7f1d27 wrote the change into the v0.97.0 entry and 79e31ac bumped the manifest to 0.97.1, so a reader matching the running tag against hub/CHANGELOG.md found no v0.97.1 heading at all. Fixed in this session — the paragraph now has its own entry, marked as added retroactively. Second occurrence of the class (the first was agent 0.90.1).

---

2. Scope, and what was deliberately not isolated

Campaign 10 could run with Tier 3 OFF; a campaign about off-site recovery cannot. Off-site hard-requires the DR tier, which provisions on ep0 (Tier 2). The Phase-0 operator ruling stands and is restated here rather than quietly inherited: ep0 and the Hetzner Storage Box are written to, additively — a PBS namespace and token, a WireGuard peer, and a Storage Box sub-account, all created on the ordinary customer path. Nothing existing is modified or deleted. The brief's I7 wording ("ep0 read-only") was relaxed by that ruling, not widened by this session.

Phase 2 added no new external writes. Its faults are network blocks, a service stop, a container restart and a VM shutdown — all on the campaign's own appliance, all reverted, each with a positive control proving the fault was real.

---

3. The two positives that were owed (brief §4)

§4.1 — is the floor actually SERVED? MEASURED. YES. Twice, independently.

The previous session recorded "no HELD line and no held reason" — two absences — and correctly refused to call that a measurement. Both positives were taken here.

(a) The box's own rendered state. /settings → „Automatikus frissítés – Minimális verzió (üzemeltető) 0.200.0". That value is s.updater.GetFloor(), and u.floor has exactly one writer — SetFloor, whose only non-test caller is the report-ACK handler. Both hold branches of ResolveManagedFloor set Floor = "" (store.go:2086, :2095), pinned by managed_floor_test.go:94. A non-empty floor on the box therefore proves an ACK carried one.

(b) A cold-started process saying it out loud. F8's restart produced the decisive A/B — the same log line, same box, same code, before and after the golden rebake:

15:58:35  settle-gate: GO — floor still unknown after 1m30s               ← hold IN FORCE
21:06:16  settle-gate: GO — at/above floor 0.200.0 (we are 0.201.0)       ← hold RELEASED

The hub half, as a positive rather than an absence: managed floor HELD for c11 … fired every 15 minutes from 17:57:11 to 22:12:06 and then stopped — and the silence is backed by a liveness control (the hub logged host-report from demo-hp-bb76ea at 22:44:41 and wgsync: pushed 5 peers at 22:44:57, so it was demonstrably still logging).

A correction to the brief's own plan. It says to take this at DEBUG during F8's restart. A restart alone would not have produced it: SetFloor's line is u.dbg(...), gated on a private flag set from cfg.Logging.Level == "debug", and it writes to the logger — never to the logx debug ring, so it cannot appear in /api/debug/logs at any level. Confirmed live: 4 000 ring entries spanning the release window contain no SetFloor line, with a level census run first (1196 DEBUG / 2802 INFO / 2 WARN) so the absence was known to be structural rather than evidential.

§4.2 — R-218's live half: still NOT measured, deliberately, with the reason

The fix is present and readable — needsOffsiteCredential now retires the declaration on the target, not on the key:

if t != nil { return false }   // a TARGET exists — not a rebuild

The venue cannot exercise it. c11 has a target (applied_marker present since 14:57), so t != nil and the box correctly does not declare; declaring here would be the bug. The state that exercises the fix — hub identity blob present, key placed, no target — is shape (a), which the venue held during Phase 1 and does not hold now. F7's set-aside was examined as a route to it and does not produce it either (§5, F7).

Recorded as still not measured rather than inferred from the unit test. It needs one rebuild, which the brief forbids before Phase 4.

---

4. Phase 2 — the eleven faults

Judged on the message, not the outcome. Every fault carries a positive control proving the fault was real; every control that failed is reported as a failed control, not as a result.

The four messages (internal/web/recovery_handlers.go, v0.201.0)

	fires when	first words
M1	unseal failed and no superseded package	„A megadott helyreállítási kódot nem fogadtuk el. Ellenőrizd, hogy mind a tíz szót…"
M2	the capability gate says the agent cannot do it (R-216)	„Ez a gép még nem tudja megnyitni a mentéseidet… A kódoddal semmi baj…"
M3	unlocked, inventory unreadable (R-217)	„A kulcs visszakerült, de a mentések listáját most nem sikerült beolvasni…"
M4	unseal failed and a superseded package is retained (R-222)	„Ez a kód nem nyitja meg azt a csomagot, amit most őrzünk ehhez a géphez…"

The venue retains a superseded package, so M4's branch — tested before M1 on the failure path — is live for every failed unlock. That single ordering fact produces two of the four new findings.

Results

	Fault	Right answer	Result
F1	wrong code ×3	refused, M1 only, no lockout, nothing written	PARTIAL — refused ✅, no lockout ✅, nothing written ✅, but M4, not M1 → R-226
F2	no code at all	states nobody can recover it; offers set-aside	PASS
F3	hub unreachable	names the hub, never the code	FAIL — M4. → R-224
F4	agent stopped	M2, the R-216 fix under pressure	FAIL — M4. → R-224
F5	store unreachable after a successful unlock	unlock counts; „could not be read", not „opened with content"	PASS — R-217's fix holds
F6	„Most nem", return later	entry point survives; unlock still works	PASS
F7	set aside, then a change of mind	set aside, never deleted; honest afterwards	(see §4.7)
F8	controller restarted mid-unlock	no half-state; the screen says which	PARTIAL — no half-state ✅, but a raw English Bad Gateway → R-227
F9	a box that never had off-site	nothing, incl. a direct GET /recovery	(see §4.9)
F10	app with a mandatory data path missing	run reports incomplete, names the app, digest carries it	(see §4.10)
F11	box offline a whole reporting window	hub and box agree once it returns	(see §4.11)

(F7, F9, F10 and F11 are written up in the sections below; the table is the index.)

The timing discriminator used throughout

age's scrypt makes a real unseal cost ~1 s. Phase 1's headline was diagnosed by a 0.134 s response, and the same instrument separates every fault below:

	elapsed	what it means
F1 wrong code	1.194 / 1.004 / 1.014 s	a real unseal was attempted and failed
F3 hub down	0.0556 s	no unseal attempted — it failed before the KDF
F4 agent down	0.0299 s	no unseal attempted
F5 store down	1.198 s	a real unseal, which SUCCEEDED

The customer sees the same sentence for the 1.0 s case and the 0.03 s case.

---

5. Findings

R-224 — every non-code failure on the unlock path is reported as a statement about the code

F3 and F4 are one defect with two faces, and it is Phase 1's headline finding relocated from the version channel to the transport.

	injected (control)	customer sees	machine's own log
F3	hub REJECTed (302 → exit 7)	M4	fetching the sealed bundle: hub: transport error: … no route to host
F4	systemctl stop felhom-agent (:8443 gone)	M4	dial tcp 169.254.253.1:8443: connect: connection refused

In both the correct, current recovery code was entered, so the only possible cause of failure was the injected fault.

The discriminator exists and is discarded at the HTTP boundary. The agent's own err separates the cases exactly —

wrong code : "escrow: the recovery code did not unwrap the identity escrow …"
hub down   : "escrow: fetching the sealed bundle: hub: transport error: … no route to host"

— but both return HTTP 400 under one merged sentence ("the recovery code did not open the sealed bundle, or the bundle could not be fetched"), the agent's own msg= collapses them too, and the controller's failure path has no branch for "could not ask / could not reach" at all. rerr != nil falls straight into the code/package messages.

Why R-216's gate did not catch F4, measured rather than reasoned — from the box's debug ring:

[web] recovery capability gate: offsite_key_recovery=yes (source=version)

source=version. The gate answers from the known agent version without probe traffic, which is correct for the question it was built for (is this agent too old?) and cannot answer the question it is being used for (can this machine ask right now?). A dead agent of the right version sails through it — and the failure that follows is attributed to the code, which R-216's own comment says must never happen: "An attempt that cannot succeed must never be made, because its failure is attributed to the code."

Consequence. During any hub outage or agent restart, a customer holding a perfect recovery code is told it does not open their package and is routed to customer support about older backups. I6 — an unreachable service reported as a fact about the code.

R-225 — the store reports 0 snapshots · 0 GB when it cannot read it, beside a card saying it holds backups

Found while checking F6's "the listing is coherent" clause. /backups/remote renders, on one screen:

Tároló méret · 0 pillanatkép            Tárhelykeret: 0 / 50 GB (0%)

immediately above:

„A távoli tároló másik kulccsal készült mentéseket tartalmaz. … A meglévő mentések nem sérültek…"

Ground truth, measured directly against the Storage Box over SFTP — a read-only listing, no decryption, using the box's own transport credential:

/home/felhom-repo/snapshots:
  f3d9cd67d539c00359f0454ea7a782f45beac8406c55275bee5a6886afa8d791   (Aug 5 13:14)
/home/felhom-repo:   du -s → 12535 KB          /home/felhom-repo/keys:   exactly ONE key

That is the Phase 0 snapshot holding all three sentinels — the customer's only surviving copy — matching the journal's repo_size_bytes 12 611 522.

Mechanism, from the box's own state: after the Phase 3 rebuild the offbox block in settings.json carries no snapshot_count and no repo_size_bytes key at all. The values are unknown, and unknown renders as the zero value.

This is R-217's defect class in a second location — a field whose zero is indistinguishable from a real measurement, defaulted past on an unknown path. OffsiteInventory.Empty exists precisely because "len(Apps)==0 is also what a failed read looks like". I6.

I5 checked and NOT breached: the hub's /offsite shows Campaign 11 · 0.0 GB, but that is 12.5 MB rounded to one decimal of a GB and the pool total (Used 3.8 GB) is consistent. The two views do not disagree; both understate, for different reasons, and only the box's snapshot count — an integer — is false.

R-226 — M1, the only message that tells a customer to check their typing, is unreachable on any box that has re-escrowed

The failure path tests M4's condition before M1's:

if present, at := s.recoverySuperseded(); present { …M4…; return }
…M1…

So on every box the hub keeps an earlier package for, a genuinely mistyped code produces M4 — which is hedged („Ha egy korábbi kódot adtál meg") and states two true facts, but offers no hint to re-check the ten words and routes the customer to support about older backups.

Measured: F1's three wrong-code attempts each returned M4, each after a real ~1 s unseal.

Why it matters rather than being a nicety: the population that has re-escrowed is exactly the population that has just been handed a new recovery code and is most likely to be typing one. R-222's fix removed one conflation (correct-earlier-code read as a mistype) and introduced another (mistype read as a correct-earlier-code) on the same branch.

R-227 — a restart mid-unlock returns a raw English Bad Gateway

F8 restarted the controller at T+0.7 s, inside the unseal window (control: the container's StartedAt moved). The customer got:

HTTP 502 · "Bad Gateway"

A raw upstream error, in English, from traefik. It names no reason, offers no action, and says nothing about whether the key was installed. I3 — "every refusal names a reason a person can act on, in Hungarian, with no raw error." Low severity (the window is ~1 s wide) and recorded rather than inflated.

---

6. What behaved correctly

A campaign that reports only what broke is half a campaign. These were tested and held.

R-217's fix	F5: with the store blocked after a successful unlock, the page rendered M3 and no listing block at all. The four false-claim strings („A tároló megnyílt", „van benne tartalom", „nem tudtuk alkalmazásokhoz rendelni") are absent — verified in UTF-8 with two accented positive controls present, after the documented accented-substring false-zero trap was accounted for
No lockout	F1: three wrong codes, three identical responses, no rate limit, no refusal to try again — as §8.4 NO LOCKOUT documents
Nothing written on failure	F1: all four /data/offbox files byte-identical, mtimes frozen at 14:57:35
I4	the entered code appears in no log, file or page. Sweep run with a planted-canary control first; the only hits were the harness's own script
The „Most nem" asymmetry	F6: the full page stops interrupting /launcher and /dashboard, while /recovery stays 200 and the backups-area entry point survives — bound to the offer, never to the postpone flag
No half-state	F8: a restart mid-unlock left the key files and settings.json coherent; the controller returned healthy in 40 s
R-215's fix	the GET /recovery gate is present in v0.201.0 and consults the same predicate as the POST sibling
R-198's retention, in the UI	the hub host page reads „Key Escrow: present · 1 superseded escrow blob(s) retained"
The empty submission	F2: „Add meg a helyreállítási kódot." in 0.027 s — no unseal, and no blame attached to a code never given

---

7. Harness faults, separated from the product's

Four, all mine, none a product defect — recorded because Campaign 10's §4d is the format and because two of them nearly produced false findings.

1. source ~/.config/credentials echoes secrets. The file holds keys with hyphens (R_DEMO-HP=…) that bash cannot assign, and the command not found error prints the value. Two demo-box recovery codes were printed this way before the helper was rewritten to grep the one key it needs. A live I4 hazard for any session that sources that file — worth a memory, not a register row.
2. Quoted values in that file — a bare cut -d= keeps the quotes and yields the wrong secret; the hub returned 302 until they were stripped. Already in project memory; re-confirmed.
3. The recovery page carries no <meta name="csrf-token"> — it renders a hidden _csrf input (data["CSRFField"]). The first F1 run produced three 403s and CSRF-LEN-0; the harness's own length check caught it, and the controller's log named the reason word (token mismatch). Harness, not product.
4. Two failed reachability controls, reported as failures. The F5 probe first used nc, which the controller container does not have, so blocked and unblocked printed the same fallback; the second attempt used the IPv6 address getent hosts prefers, and the container has no IPv6 route at all. Only a /dev/tcp probe against the IPv4 address (91.98.242.176) discriminated TCP-OPEN → TCP-CLOSED. Two readings that looked like results and were instrument failures.

---

8. Suspicions investigated

	verdict
"The floor is still held — there is no HELD line and no held reason"	DISPROVED. Both absences were real, and both are explained: the hub stops logging when it stops holding, and SetFloor's DEBUG line cannot reach the debug ring at all. The floor is served, measured two ways (§3)
"The hub and the box disagree about the store's contents (I5)"	DISPROVED. Both understate; the hub's 0.0 GB is rounding of 12.5 MB. Only the box's snapshot count is false → R-225, which is an I6 finding, not an I5 one
"GET /recovery still renders on a box it should not"	CONFIRMED FIXED — the gate is present in v0.201.0 (§9, F9)

---

(Sections 9–12 — F7, F9, F10, F11, Phase 4, invariants, teardown and hygiene — follow below.)