CAMPAIGN-11 Phases 2+4: the fault journal, the campaign document, and hub v0.97.1's missing heading
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
Phase 2 (eleven injected faults) and the §4 positives that were owed. - §4.1 MEASURED, twice: the box's rendered GetFloor() is 0.200.0, and a cold-started controller logs "settle-gate: GO — at/above floor 0.200.0" against the same line reading "floor still unknown" while the hold was in force. Also corrects the brief's plan: SetFloor's line is u.dbg(), gated on logging.level=debug and written to the logger, so it can NEVER reach the debug ring — a restart alone would not have produced it. - §4.2 still NOT measured, deliberately: the venue has an off-site target, so needsOffsiteCredential correctly returns false. Recorded, not inferred from the unit test. - F1 PARTIAL, F2 PASS, F3 FAIL, F4 FAIL, F5 PASS, F6 PASS, F8 PARTIAL. F3+F4: a hub outage and a stopped agent are both rendered as "this code does not open your package", in 0.056 s and 0.030 s — no unseal attempted. The agent's own err field distinguishes them exactly and it is discarded at the HTTP boundary; the R-216 capability gate answers source=version and cannot see reachability. - R-217's fix HOLDS under exactly its fault (F5), verified with the false-claim strings absent and accented positive controls present. hub/CHANGELOG.md: v0.97.1 had no heading of its own — the change was written into the v0.97.0 entry while the deployed tag is 0.97.1. Given its own entry, marked as added retroactively. Second occurrence of the class (agent 0.90.1). Evidence: documentation/tests/campaign11-evidence-2026-08-05/journal-phase24.md No product code changed.
This commit is contained in:
@@ -0,0 +1,354 @@
|
||||
# CAMPAIGN 11 — the recovery journey (2026-08-05 → 08-06)
|
||||
|
||||
**Four phases. Phase 1 (the clean journey) FAILED on the journey and PASSED on the data. Phase 3 (key
|
||||
supersession) PASSED on its central question. Phase 2 (eleven injected faults) and Phase 4 (an
|
||||
unattended soak) ran overnight on 2026-08-05/06 and are reported here for the first time.**
|
||||
|
||||
**Findings: R-214 … R-223 from Phases 1 and 3** (seven since fixed and shipped), **plus R-224 … R-227
|
||||
from Phase 2.** **Three suspicions investigated — two confirmed, one DISPROVED.** **Four harness
|
||||
faults, separated from the product's.**
|
||||
|
||||
Evidence: `../tests/campaign11-evidence-2026-08-05/` — `journal.md` (Phases 0, 1, 3) and
|
||||
`journal-phase24.md` (Phases 2 and 4, every observable in the order taken).
|
||||
|
||||
> **The one-line answer to the question this campaign was built to ask.** A Hungarian household whose
|
||||
> machine is rebuilt **gets their data back only if an operator is standing next to them.** The
|
||||
> cryptography, the retention and the transport all work and are now proven live. **What fails is
|
||||
> being told the truth**: on this box, four different situations — a mistyped code, a hub outage, a
|
||||
> stopped agent, and a correct code for a retained earlier package — produce **one** message, and
|
||||
> three of the four are wrong.
|
||||
|
||||
---
|
||||
|
||||
## 1. Venue and baselines
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host | `demo-hp` (HP t740), **Tier 0**, the designated drill host. Reached **by SSH key, first try** — R-129 stands |
|
||||
| VM | **321 `c11-appliance`** — q35/OVMF, 4 cores, 8 GB, `cpu=host` |
|
||||
| Disks | `scsi0` 200 G system · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` |
|
||||
| Storage | **`c11-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT** (a subdirectory fails the agent's `exactMount` check; Campaign 10 §1) |
|
||||
| Appliance | `192.168.0.105`, hostname `c11`, `felhom-agent 0.125.0` |
|
||||
| Guest | LXC **9201**, **`192.168.0.106`** — DHCP, and it MOVED between phases (`.207` → `.227` → `.106`) |
|
||||
| Hub customer | **`c11`** "Campaign 11", domain `c11.felhom.eu`, DR tier ON, off-site ON |
|
||||
| Host id | **`c11-36d660`** |
|
||||
| Untouched | `drill-r50` (VM 300, stopped), guest 9201 on both demo boxes, `local-lvm`, `felhom-backup`, every other hub customer |
|
||||
|
||||
### Baselines — every value re-read fresh at the start of Phase 2
|
||||
|
||||
| What | Value | How |
|
||||
|---|---|---|
|
||||
| `felhom-controller` `main` | **v0.201.0** @ `05cf352a2f17` | `HEAD` == `origin/main`, tree clean |
|
||||
| `felhom-agent` `main` | **v0.125.0** @ `0404f60e6a7b` | same |
|
||||
| `felhom.eu` `main` | @ `3a539ea5306a` | same |
|
||||
| hub, LIVE | **`felhom-hub:0.97.1`** | `kubectl … get deploy hub -o jsonpath` |
|
||||
| Day-0 manifest | agent **0.125.0** · golden **0.201.0** · `min_agent` **0.125.0** | hub `/configuration`, `selected` options |
|
||||
| c11 floor | **`v0.200.0 (override)`**; every other customer `v0.156.0` | hub `/configs` |
|
||||
| Highest register ID | **R-223** | fresh `grep -rhoE 'R-[0-9]{1,3}'` over all four repos |
|
||||
|
||||
**All four cited commits match the brief exactly**, and the manifest matches the golden rebake the
|
||||
Phase-1/3 journal records. **Blast radius of the per-customer floor is still zero, measured** — the
|
||||
other five customers all read `v0.156.0`.
|
||||
|
||||
### The hub CHANGELOG/deployed mismatch — diagnosed more precisely than filed
|
||||
|
||||
The brief records *"`0.97.1` shipped without an entry"*. **The content was not missing; its heading
|
||||
was.** Commit `a7f1d27` wrote the change into the **v0.97.0** entry and `79e31ac` bumped the manifest
|
||||
to `0.97.1`, so a reader matching the running tag against `hub/CHANGELOG.md` found no `v0.97.1`
|
||||
heading at all. **Fixed in this session** — the paragraph now has its own entry, marked as added
|
||||
retroactively. Second occurrence of the class (the first was agent `0.90.1`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Scope, and what was deliberately not isolated
|
||||
|
||||
Campaign 10 could run with Tier 3 OFF; **a campaign about off-site recovery cannot.** Off-site
|
||||
hard-requires the DR tier, which provisions on **ep0** (Tier 2). The Phase-0 operator ruling stands
|
||||
and is restated here rather than quietly inherited: **ep0 and the Hetzner Storage Box are written to,
|
||||
additively** — a PBS namespace and token, a WireGuard peer, and a Storage Box sub-account, all created
|
||||
on the ordinary customer path. **Nothing existing is modified or deleted.** The brief's I7 wording
|
||||
("ep0 read-only") was relaxed by that ruling, not widened by this session.
|
||||
|
||||
**Phase 2 added no new external writes.** Its faults are network blocks, a service stop, a container
|
||||
restart and a VM shutdown — all on the campaign's own appliance, all reverted, each with a positive
|
||||
control proving the fault was real.
|
||||
|
||||
---
|
||||
|
||||
## 3. The two positives that were owed (brief §4)
|
||||
|
||||
### §4.1 — is the floor actually SERVED? **MEASURED. YES. Twice, independently.**
|
||||
|
||||
The previous session recorded *"no HELD line and no held reason"* — **two absences** — and correctly
|
||||
refused to call that a measurement. Both positives were taken here.
|
||||
|
||||
**(a) The box's own rendered state.** `/settings` → „Automatikus frissítés – **Minimális verzió
|
||||
(üzemeltető) `0.200.0`**". That value is `s.updater.GetFloor()`, and `u.floor` has exactly one writer
|
||||
— `SetFloor`, whose only non-test caller is the report-ACK handler. **Both hold branches of
|
||||
`ResolveManagedFloor` set `Floor = ""`** (`store.go:2086`, `:2095`), pinned by
|
||||
`managed_floor_test.go:94`. A non-empty floor on the box therefore proves an ACK carried one.
|
||||
|
||||
**(b) A cold-started process saying it out loud.** F8's restart produced the decisive A/B — the same
|
||||
log line, same box, same code, before and after the golden rebake:
|
||||
|
||||
```
|
||||
15:58:35 settle-gate: GO — floor still unknown after 1m30s ← hold IN FORCE
|
||||
21:06:16 settle-gate: GO — at/above floor 0.200.0 (we are 0.201.0) ← hold RELEASED
|
||||
```
|
||||
|
||||
**The hub half, as a positive rather than an absence:** `managed floor HELD for c11 …` fired every
|
||||
15 minutes from 17:57:11 to **22:12:06** and then stopped — and the silence is backed by a liveness
|
||||
control (the hub logged `host-report from demo-hp-bb76ea` at 22:44:41 and `wgsync: pushed 5 peers` at
|
||||
22:44:57, so it was demonstrably still logging).
|
||||
|
||||
> **A correction to the brief's own plan.** It says to take this at DEBUG during F8's restart.
|
||||
> **A restart alone would not have produced it**: `SetFloor`'s line is `u.dbg(...)`, gated on a private
|
||||
> flag set from `cfg.Logging.Level == "debug"`, and it writes to the logger — **never to the logx debug
|
||||
> ring**, so it cannot appear in `/api/debug/logs` at any level. Confirmed live: 4 000 ring entries
|
||||
> spanning the release window contain no `SetFloor` line, **with a level census run first** (1196
|
||||
> DEBUG / 2802 INFO / 2 WARN) so the absence was known to be structural rather than evidential.
|
||||
|
||||
### §4.2 — R-218's live half: **still NOT measured, deliberately, with the reason**
|
||||
|
||||
The fix is present and readable — `needsOffsiteCredential` now retires the declaration on the
|
||||
**target**, not on the key:
|
||||
|
||||
```go
|
||||
if t != nil { return false } // a TARGET exists — not a rebuild
|
||||
```
|
||||
|
||||
**The venue cannot exercise it.** c11 has a target (`applied_marker` present since 14:57), so
|
||||
`t != nil` and the box correctly does **not** declare; declaring here would be the bug. The state that
|
||||
exercises the fix — hub identity blob present, key placed, **no target** — is shape (a), which the
|
||||
venue held during Phase 1 and does not hold now. F7's set-aside was examined as a route to it and
|
||||
**does not produce it either** (§5, F7).
|
||||
|
||||
**Recorded as still not measured rather than inferred from the unit test.** It needs one rebuild,
|
||||
which the brief forbids before Phase 4.
|
||||
|
||||
---
|
||||
|
||||
## 4. Phase 2 — the eleven faults
|
||||
|
||||
**Judged on the message, not the outcome.** Every fault carries a positive control proving the fault
|
||||
was real; **every control that failed is reported as a failed control, not as a result.**
|
||||
|
||||
### The four messages (`internal/web/recovery_handlers.go`, v0.201.0)
|
||||
|
||||
| | fires when | first words |
|
||||
|---|---|---|
|
||||
| **M1** | unseal failed **and** no superseded package | „A megadott helyreállítási kódot **nem fogadtuk el**. Ellenőrizd, hogy mind a tíz szót…" |
|
||||
| **M2** | the capability gate says the agent cannot do it (R-216) | „Ez a gép **még nem tudja megnyitni** a mentéseidet… **A kódoddal semmi baj**…" |
|
||||
| **M3** | unlocked, inventory unreadable (R-217) | „**A kulcs visszakerült**, de a mentések listáját most nem sikerült beolvasni…" |
|
||||
| **M4** | unseal failed **and** a superseded package is retained (R-222) | „**Ez a kód nem nyitja meg azt a csomagot, amit most őrzünk** ehhez a géphez…" |
|
||||
|
||||
**The venue retains a superseded package**, so M4's branch — tested **before** M1 on the failure path
|
||||
— is live for every failed unlock. That single ordering fact produces two of the four new findings.
|
||||
|
||||
### Results
|
||||
|
||||
| | Fault | Right answer | Result |
|
||||
|---|---|---|---|
|
||||
| **F1** | wrong code ×3 | refused, **M1 only**, no lockout, nothing written | **PARTIAL** — refused ✅, no lockout ✅, nothing written ✅, but **M4, not M1** → **R-226** |
|
||||
| **F2** | no code at all | states nobody can recover it; offers set-aside | **PASS** |
|
||||
| **F3** | hub unreachable | names **the hub**, never the code | **FAIL** — M4. → **R-224** |
|
||||
| **F4** | agent stopped | **M2**, the R-216 fix under pressure | **FAIL** — M4. → **R-224** |
|
||||
| **F5** | store unreachable after a successful unlock | unlock counts; „could not be read", not „opened with content" | **PASS** — R-217's fix holds |
|
||||
| **F6** | „Most nem", return later | entry point survives; unlock still works | **PASS** |
|
||||
| **F7** | set aside, then a change of mind | set aside, never deleted; honest afterwards | *(see §4.7)* |
|
||||
| **F8** | controller restarted mid-unlock | no half-state; the screen says which | **PARTIAL** — no half-state ✅, but a raw English `Bad Gateway` → **R-227** |
|
||||
| **F9** | a box that never had off-site | nothing, incl. a direct `GET /recovery` | *(see §4.9)* |
|
||||
| **F10** | app with a mandatory data path missing | run reports `incomplete`, names the app, digest carries it | *(see §4.10)* |
|
||||
| **F11** | box offline a whole reporting window | hub and box agree once it returns | *(see §4.11)* |
|
||||
|
||||
*(F7, F9, F10 and F11 are written up in the sections below; the table is the index.)*
|
||||
|
||||
### The timing discriminator used throughout
|
||||
|
||||
`age`'s scrypt makes a real unseal cost ~1 s. **Phase 1's headline was diagnosed by a 0.134 s
|
||||
response**, and the same instrument separates every fault below:
|
||||
|
||||
| | elapsed | what it means |
|
||||
|---|---|---|
|
||||
| F1 wrong code | **1.194 / 1.004 / 1.014 s** | a real unseal was attempted and failed |
|
||||
| F3 hub down | **0.0556 s** | **no unseal attempted** — it failed before the KDF |
|
||||
| F4 agent down | **0.0299 s** | **no unseal attempted** |
|
||||
| F5 store down | **1.198 s** | a real unseal, which SUCCEEDED |
|
||||
|
||||
**The customer sees the same sentence for the 1.0 s case and the 0.03 s case.**
|
||||
|
||||
---
|
||||
|
||||
## 5. Findings
|
||||
|
||||
### R-224 — every non-code failure on the unlock path is reported as a statement about the code
|
||||
|
||||
**F3 and F4 are one defect with two faces**, and it is Phase 1's headline finding relocated from the
|
||||
version channel to the transport.
|
||||
|
||||
| | injected (control) | customer sees | machine's own log |
|
||||
|---|---|---|---|
|
||||
| **F3** | hub REJECTed (`302` → `exit 7`) | **M4** | `fetching the sealed bundle: hub: transport error: … no route to host` |
|
||||
| **F4** | `systemctl stop felhom-agent` (`:8443` gone) | **M4** | `dial tcp 169.254.253.1:8443: connect: connection refused` |
|
||||
|
||||
In both the **correct, current** recovery code was entered, so the only possible cause of failure was
|
||||
the injected fault.
|
||||
|
||||
**The discriminator exists and is discarded at the HTTP boundary.** The agent's own `err` separates
|
||||
the cases exactly —
|
||||
|
||||
```
|
||||
wrong code : "escrow: the recovery code did not unwrap the identity escrow …"
|
||||
hub down : "escrow: fetching the sealed bundle: hub: transport error: … no route to host"
|
||||
```
|
||||
|
||||
— but both return **HTTP 400** under one merged sentence (*"the recovery code did not open the sealed
|
||||
bundle, **or the bundle could not be fetched**"*), the agent's own `msg=` collapses them too, and the
|
||||
controller's failure path has **no branch for "could not ask / could not reach"** at all. `rerr != nil`
|
||||
falls straight into the code/package messages.
|
||||
|
||||
**Why R-216's gate did not catch F4**, measured rather than reasoned — from the box's debug ring:
|
||||
|
||||
```
|
||||
[web] recovery capability gate: offsite_key_recovery=yes (source=version)
|
||||
```
|
||||
|
||||
**`source=version`.** The gate answers from the *known agent version* without probe traffic, which is
|
||||
correct for the question it was built for (*is this agent too old?*) and cannot answer the question it
|
||||
is being used for (*can this machine ask right now?*). **A dead agent of the right version sails
|
||||
through it** — and the failure that follows is attributed to the code, which R-216's own comment says
|
||||
must never happen: *"An attempt that cannot succeed must never be made, because its failure is
|
||||
attributed to the code."*
|
||||
|
||||
**Consequence.** During any hub outage or agent restart, a customer holding a perfect recovery code is
|
||||
told it does not open their package and is routed to customer support about *older* backups. **I6** —
|
||||
an unreachable service reported as a fact about the code.
|
||||
|
||||
### R-225 — the store reports `0 snapshots · 0 GB` when it cannot read it, beside a card saying it holds backups
|
||||
|
||||
Found while checking F6's *"the listing is coherent"* clause. `/backups/remote` renders, **on one
|
||||
screen**:
|
||||
|
||||
```
|
||||
Tároló méret · 0 pillanatkép Tárhelykeret: 0 / 50 GB (0%)
|
||||
```
|
||||
|
||||
immediately above:
|
||||
|
||||
> „**A távoli tároló másik kulccsal készült mentéseket tartalmaz.** … **A meglévő mentések nem
|
||||
> sérültek**…"
|
||||
|
||||
**Ground truth, measured directly against the Storage Box over SFTP** — a read-only listing, no
|
||||
decryption, using the box's own transport credential:
|
||||
|
||||
```
|
||||
/home/felhom-repo/snapshots:
|
||||
f3d9cd67d539c00359f0454ea7a782f45beac8406c55275bee5a6886afa8d791 (Aug 5 13:14)
|
||||
/home/felhom-repo: du -s → 12535 KB /home/felhom-repo/keys: exactly ONE key
|
||||
```
|
||||
|
||||
That is the Phase 0 snapshot holding **all three sentinels** — the customer's only surviving copy —
|
||||
matching the journal's `repo_size_bytes 12 611 522`.
|
||||
|
||||
**Mechanism, from the box's own state:** after the Phase 3 rebuild the `offbox` block in
|
||||
`settings.json` carries **no `snapshot_count` and no `repo_size_bytes` key at all**. The values are
|
||||
*unknown*, and unknown renders as the zero value.
|
||||
|
||||
**This is R-217's defect class in a second location** — a field whose zero is indistinguishable from a
|
||||
real measurement, defaulted past on an unknown path. `OffsiteInventory.Empty` exists precisely because
|
||||
*"len(Apps)==0 is also what a failed read looks like"*. **I6.**
|
||||
|
||||
**I5 checked and NOT breached:** the hub's `/offsite` shows `Campaign 11 · 0.0 GB`, but that is 12.5 MB
|
||||
rounded to one decimal of a GB and the pool total (`Used 3.8 GB`) is consistent. The two views do not
|
||||
disagree; **both understate, for different reasons**, and only the box's snapshot **count** — an
|
||||
integer — is false.
|
||||
|
||||
### R-226 — M1, the only message that tells a customer to check their typing, is unreachable on any box that has re-escrowed
|
||||
|
||||
The failure path tests M4's condition **before** M1's:
|
||||
|
||||
```go
|
||||
if present, at := s.recoverySuperseded(); present { …M4…; return }
|
||||
…M1…
|
||||
```
|
||||
|
||||
So on every box the hub keeps an earlier package for, a **genuinely mistyped code** produces M4 —
|
||||
which is hedged (*„**Ha** egy korábbi kódot adtál meg"*) and states two true facts, but offers no hint
|
||||
to re-check the ten words and routes the customer to support about older backups.
|
||||
|
||||
**Measured:** F1's three wrong-code attempts each returned M4, each after a real ~1 s unseal.
|
||||
|
||||
**Why it matters rather than being a nicety:** the population that has re-escrowed is exactly the
|
||||
population that has just been handed a new recovery code and is most likely to be typing one. R-222's
|
||||
fix removed one conflation (correct-earlier-code read as a mistype) and introduced another
|
||||
(mistype read as a correct-earlier-code) **on the same branch**.
|
||||
|
||||
### R-227 — a restart mid-unlock returns a raw English `Bad Gateway`
|
||||
|
||||
F8 restarted the controller at T+0.7 s, inside the unseal window (control: the container's `StartedAt`
|
||||
moved). The customer got:
|
||||
|
||||
```
|
||||
HTTP 502 · "Bad Gateway"
|
||||
```
|
||||
|
||||
A raw upstream error, in **English**, from traefik. It names no reason, offers no action, and says
|
||||
nothing about whether the key was installed. **I3** — *"every refusal names a reason a person can act
|
||||
on, in Hungarian, with no raw error."* Low severity (the window is ~1 s wide) and recorded rather than
|
||||
inflated.
|
||||
|
||||
---
|
||||
|
||||
## 6. What behaved correctly
|
||||
|
||||
**A campaign that reports only what broke is half a campaign.** These were tested and held.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **R-217's fix** | F5: with the store blocked after a successful unlock, the page rendered M3 and **no listing block at all**. The four false-claim strings („A tároló megnyílt", „van benne tartalom", „nem tudtuk alkalmazásokhoz rendelni") are absent — verified in UTF-8 with two accented positive controls present, after the documented accented-substring false-zero trap was accounted for |
|
||||
| **No lockout** | F1: three wrong codes, three identical responses, no rate limit, no refusal to try again — as `§8.4 NO LOCKOUT` documents |
|
||||
| **Nothing written on failure** | F1: all four `/data/offbox` files byte-identical, **mtimes frozen at 14:57:35** |
|
||||
| **I4** | the entered code appears in **no** log, file or page. Sweep run with a planted-canary control first; the only hits were the harness's own script |
|
||||
| **The „Most nem" asymmetry** | F6: the full page stops interrupting `/launcher` and `/dashboard`, while `/recovery` stays 200 and the backups-area entry point survives — bound to the offer, never to the postpone flag |
|
||||
| **No half-state** | F8: a restart mid-unlock left the key files and `settings.json` coherent; the controller returned healthy in 40 s |
|
||||
| **R-215's fix** | the `GET /recovery` gate is present in v0.201.0 and consults the same predicate as the POST sibling |
|
||||
| **R-198's retention, in the UI** | the hub host page reads „Key Escrow: present · **1 superseded escrow blob(s) retained**" |
|
||||
| **The empty submission** | F2: „Add meg a helyreállítási kódot." in 0.027 s — no unseal, and no blame attached to a code never given |
|
||||
|
||||
---
|
||||
|
||||
## 7. Harness faults, separated from the product's
|
||||
|
||||
**Four, all mine, none a product defect** — recorded because Campaign 10's §4d is the format and
|
||||
because two of them nearly produced false findings.
|
||||
|
||||
1. **`source ~/.config/credentials` echoes secrets.** The file holds keys with hyphens
|
||||
(`R_DEMO-HP=…`) that bash cannot assign, and the `command not found` error **prints the value**.
|
||||
Two demo-box recovery codes were printed this way before the helper was rewritten to `grep` the one
|
||||
key it needs. **A live I4 hazard for any session that sources that file** — worth a memory, not a
|
||||
register row.
|
||||
2. **Quoted values in that file** — a bare `cut -d=` keeps the quotes and yields the wrong secret; the
|
||||
hub returned `302` until they were stripped. Already in project memory; re-confirmed.
|
||||
3. **The recovery page carries no `<meta name="csrf-token">`** — it renders a hidden `_csrf` input
|
||||
(`data["CSRFField"]`). The first F1 run produced three `403`s and `CSRF-LEN-0`; **the harness's own
|
||||
length check caught it**, and the controller's log named the reason word (`token mismatch`).
|
||||
Harness, not product.
|
||||
4. **Two failed reachability controls, reported as failures.** The F5 probe first used `nc`, which the
|
||||
controller container does not have, so blocked and unblocked printed the same fallback; the second
|
||||
attempt used the IPv6 address `getent hosts` prefers, and **the container has no IPv6 route at
|
||||
all**. Only a `/dev/tcp` probe against the IPv4 address (`91.98.242.176`) discriminated
|
||||
`TCP-OPEN` → `TCP-CLOSED`. **Two readings that looked like results and were instrument failures.**
|
||||
|
||||
---
|
||||
|
||||
## 8. Suspicions investigated
|
||||
|
||||
| | verdict |
|
||||
|---|---|
|
||||
| *"The floor is still held — there is no HELD line and no held reason"* | **DISPROVED.** Both absences were real, and both are explained: the hub stops logging when it stops holding, and `SetFloor`'s DEBUG line cannot reach the debug ring at all. The floor is served, measured two ways (§3) |
|
||||
| *"The hub and the box disagree about the store's contents (I5)"* | **DISPROVED.** Both understate; the hub's `0.0 GB` is rounding of 12.5 MB. Only the box's snapshot **count** is false → R-225, which is an I6 finding, not an I5 one |
|
||||
| *"`GET /recovery` still renders on a box it should not"* | **CONFIRMED FIXED** — the gate is present in v0.201.0 (§9, F9) |
|
||||
|
||||
---
|
||||
|
||||
*(Sections 9–12 — F7, F9, F10, F11, Phase 4, invariants, teardown and hygiene — follow below.)*
|
||||
@@ -0,0 +1,459 @@
|
||||
# CAMPAIGN 11 — Phases 2 and 4 journal
|
||||
|
||||
Every observable, in the order taken. **Unattended session, started 2026-08-05 22:38 CEST.**
|
||||
|
||||
Clocks, stated once because three are in play and they differ:
|
||||
**demo-hp and the appliance = CEST · the guest (LXC 9201) = UTC · the hub = CEST** (the hub's own log
|
||||
lines read `22:12:06` at a wall-clock of `22:48`, so the hub logs local time, *not* UTC as the
|
||||
project memory's "hub=UTC" note says — see the hygiene section).
|
||||
|
||||
---
|
||||
|
||||
## Pre-flight — baselines re-read on arrival
|
||||
|
||||
| What | Value | How read |
|
||||
|---|---|---|
|
||||
| `felhom-controller` `main` | **v0.201.0** @ `05cf352a2f17` | `git rev-parse HEAD` == `origin/main`, tree clean |
|
||||
| `felhom-agent` `main` | **v0.125.0** @ `0404f60e6a7b` | same |
|
||||
| `felhom.eu` `main` | @ `3a539ea5306a` | same |
|
||||
| hub, LIVE | `gitea.dooplex.hu/admin/felhom-hub:0.97.1` | `kubectl -n felhom-system get deploy hub -o jsonpath=…` |
|
||||
| Day-0 manifest | agent **0.125.0** (`f7d8339b…7016b3`) · golden **0.201.0** (`e730d7ca…f007654`) · `min_agent` **0.125.0** | hub `/configuration`, the `selected` options |
|
||||
| Highest register ID | **R-223** | fresh `grep -rhoE 'R-[0-9]{1,3}' --include=*.md` over all four repos |
|
||||
|
||||
**All four cited commits match the brief exactly.** The manifest matches the journal's rebake record.
|
||||
|
||||
### Venue, re-confirmed alive
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host | `demo-hp` (HP t740), Tier 0. Reached **by SSH key, first try** — R-129 stands |
|
||||
| VM | **321 `c11-appliance`**, `running`, uptime of host 13 d |
|
||||
| Appliance | `192.168.0.105`, hostname `c11`, `felhom-agent 0.125.0`, guest 9201 `running` |
|
||||
| Guest | **`192.168.0.106`** (DHCP — was `.207` in Phase 0 and `.227` in Phase 1; it moves) |
|
||||
| Containers | `felhom-controller:0.201.0` (healthy) · `filebrowser` (healthy) · `traefik` |
|
||||
| Hub | `c11-36d660` **ONLINE**, agent `0.125.0`, 1/1 guests |
|
||||
| Floor | c11 `v0.200.0 (override)`; **every other customer `v0.156.0`** — blast radius still zero |
|
||||
| Untouched | `drill-r50` (VM 300, stopped), guest 9201 on both demo boxes, every other hub customer |
|
||||
|
||||
**`calibre-web` is NOT deployed.** Phase 3 destroyed the guest and never redeployed it. The venue's
|
||||
current shape is therefore *a rebuilt box that has completed a second escrow ceremony*, which is
|
||||
exactly what Phase 2's F-series needs.
|
||||
|
||||
### Harness — the access path, and two credential traps hit on the way in
|
||||
|
||||
Transport: **DooPlex → (Tailscale) → demo-hp → `192.168.0.105` (appliance) → `pct exec 9201`**, and
|
||||
for HTTP, **curl run on demo-hp**, which shares `192.168.0.0/24` with the guest. No server logic is
|
||||
skipped. **Harness, not product.** (The journal's Phase-0/1 SOCKS proxy was not needed; running the
|
||||
client on demo-hp is the same transport with fewer moving parts.)
|
||||
|
||||
Two traps, both recorded because each cost a wrong reading first:
|
||||
|
||||
1. **Never `source ~/.config/credentials`.** It holds keys with hyphens (`R_DEMO-HP=…`) that bash
|
||||
cannot assign, and the resulting `command not found` error **echoes the secret value** into the
|
||||
transcript. Two demo-box recovery codes were printed this way before the helper was rewritten to
|
||||
`grep` the single key it needs. Nothing was written to a file and the values are not reproduced
|
||||
here — but the mechanism is a live I4 hazard for any session that sources that file.
|
||||
2. **Values in that file are quoted**, so a bare `cut -d=` keeps the quotes and yields the wrong
|
||||
secret — the hub returned `302` until the quotes were stripped. Already in project memory
|
||||
(`credentials-file-values-are-quoted`); re-confirmed.
|
||||
|
||||
---
|
||||
|
||||
## §4 — the two positives that were owed
|
||||
|
||||
### §4.1 — is the floor actually SERVED? **MEASURED. YES.**
|
||||
|
||||
The previous session recorded *"no HELD line and no held reason"* — two absences — and correctly
|
||||
refused to call that a measurement. Here is the positive.
|
||||
|
||||
**The box's own view, rendered from `GetFloor()`** (`/settings`, the „Rendszer" subpage):
|
||||
|
||||
> Verzió és frissítés — Jelenlegi verzió **0.201.0** · Legújabb verzió **0.201.0 — naprakész** ·
|
||||
> Utolsó ellenőrzés `2026-08-05T15:57:36Z` · Automatikus frissítés – **Minimális verzió
|
||||
> (üzemeltető) `0.200.0` — a rendszer automatikusan erre a verzióra frissít, ha régebbi** ·
|
||||
> Utolsó frissítés Sikeres (**0.192.0 → 0.200.0**)
|
||||
|
||||
**Why that value is proof and not decoration**, established from source rather than assumed:
|
||||
|
||||
- `data["ControllerFloor"] = s.updater.GetFloor()` (`internal/web/handlers.go:1454`).
|
||||
- `u.floor` has **exactly one writer** — `SetFloor` (`selfupdate/updater.go:556`); the only
|
||||
non-test caller is the report-ACK handler `updater.SetFloor(resp.MinControllerVersion)`
|
||||
(`cmd/controller/main.go:696`).
|
||||
- **Both hold branches of `ResolveManagedFloor` set `d.Floor = ""`** (`store.go:2086` and `:2095`),
|
||||
and `managed_floor_test.go:94` pins it (`!fd.Held || fd.Floor != ""`).
|
||||
|
||||
So a **non-empty** floor on the box can only have come from an ACK that carried one. A held floor is
|
||||
served as the empty string and would render as none.
|
||||
|
||||
**The temporal half, which rules out a stale pre-hold value:**
|
||||
|
||||
| time (CEST) | observable | source |
|
||||
|---|---|---|
|
||||
| 17:57:05 | controller starts | guest log `15:57:05Z` |
|
||||
| 17:57:11 | `managed floor HELD for c11 … (controller floor withheld)` | hub log |
|
||||
| 17:58:35 | `[offsite-apply] settle-gate: GO — floor still unknown after 1m30s` | **the box's own** log |
|
||||
| 17:57 → **22:12:06**, every 15 min | `managed floor HELD for c11` ×18 | hub log |
|
||||
| after 22:12:06 | **no HELD line** | hub log |
|
||||
| 22:48 | box renders floor **`0.200.0`** | `/settings` |
|
||||
|
||||
At start the box's floor was empty **by its own account**, and the hub withheld it for the next four
|
||||
hours. The `0.200.0` it holds now can therefore only have arrived on an ACK **after the vouch
|
||||
released the hold**. That is a measurement, not arithmetic.
|
||||
|
||||
**The absence is backed by a liveness positive** (rule 6): the HELD lines stop at 22:12:06 while the
|
||||
hub demonstrably keeps logging — `22:44:41 host-report from demo-hp-bb76ea`, `22:44:57 wgsync: pushed
|
||||
5 peers`. Two further report cycles (22:27, 22:42) passed with no HELD line.
|
||||
|
||||
#### And a correction to the brief's own plan for §4.1
|
||||
|
||||
The brief says to take this positive during **F8**'s controller restart, *"at DEBUG if necessary"*.
|
||||
**A restart would not have produced it.** `SetFloor`'s log line is `u.dbg(...)`
|
||||
(`updater.go:554`), and `u.dbg` is gated on a private `u.debug` flag set from
|
||||
`cfg.Logging.Level == "debug"` (`main.go:478`) — it writes to the **logger**, never to the logx debug
|
||||
ring, so it cannot appear in `/api/debug/logs` at any level.
|
||||
|
||||
Verified live: the debug ring was pulled (4 000 entries, `17:03:08Z → 20:46:46Z` — a window that
|
||||
**covers** the 20:12→20:27 UTC release), and it contains **no** `SetFloor` line. **Positive control
|
||||
on the instrument first**: the same ring carries `1196 DEBUG / 2802 INFO / 2 WARN` entries, so the
|
||||
ring works and the level is not filtered — the absence is structural, not evidential.
|
||||
|
||||
> This is the same class the brief warns about, caught on the instrument rather than the subject: had
|
||||
> the ring been read without the level census, "no `SetFloor` line after a restart" would have looked
|
||||
> exactly like "the floor is still held".
|
||||
|
||||
**§4.1 verdict: the floor IS being served, measured from the box's own state.** Recorded against
|
||||
R-216's row.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## §4.2 — R-218's live half: deferred, with the reason
|
||||
|
||||
R-218's fix is present and readable in `backup/offbox.go:1257`:
|
||||
|
||||
```go
|
||||
func (m *Manager) needsOffsiteCredential(t *settings.OffboxTarget) bool {
|
||||
if t != nil { return false } // a TARGET exists — not a rebuild
|
||||
if m.settings == nil || !m.settings.GetHubEscrowIdentityPresent() { return false }
|
||||
return true
|
||||
}
|
||||
```
|
||||
|
||||
The short-circuit on `OffboxRepoPasswordHash()` that caused the deadlock is **gone** — the declaration
|
||||
now retires on the *target*, not on the key.
|
||||
|
||||
**The live half cannot be taken in the venue's current state**, and the reason is the fix itself: c11
|
||||
**has** a target (`host`/`user`/`port`/`repo_path` all set, `applied_marker` present since 14:57), so
|
||||
`t != nil` and the box correctly does **not** declare. Declaring here would be the bug.
|
||||
|
||||
Reaching the state that exercises it needs a box with the hub's identity blob, a placed key and **no
|
||||
target** — which the venue last held during Phase 1 and does not hold now. **Recorded as still not
|
||||
measured, deliberately, rather than reported as measured from the unit test.** F7's set-aside was
|
||||
examined as a candidate route and is reported in F7 below.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — the faults
|
||||
|
||||
Order: reversible first. **Every fault carries a positive control proving the fault was real**, and
|
||||
every control that failed is reported as a failed control rather than a result.
|
||||
|
||||
### The four messages, as identified in `internal/web/recovery_handlers.go` (v0.201.0)
|
||||
|
||||
| | trigger | first words |
|
||||
|---|---|---|
|
||||
| **M1** | the unseal failed **and** the hub holds no superseded package | „A megadott helyreállítási kódot **nem fogadtuk el**. Ellenőrizd, hogy mind a tíz szót…" |
|
||||
| **M2** | the capability gate says the agent cannot do it (R-216) | „Ez a gép **még nem tudja megnyitni** a mentéseidet… **A kódoddal semmi baj**…" |
|
||||
| **M3** | unlocked, inventory unreadable (R-217) | „A kulcs visszakerült, **de a mentések listáját most nem sikerült beolvasni**…" |
|
||||
| **M4** | the unseal failed **and** a superseded package is retained (R-222) | „**Ez a kód nem nyitja meg azt a csomagot, amit most őrzünk** ehhez a géphez…" |
|
||||
|
||||
**The venue holds a superseded package** (`hub_escrow_superseded_present: true`, `…_at 2026-08-05
|
||||
15:03:14`), so the M4 branch — which is tested *before* M1 on the failure path — is live for every
|
||||
failed unlock. That single fact shapes F1, F3 and F4 below.
|
||||
|
||||
### F1 — a wrong recovery code, three times
|
||||
|
||||
Injected: a ten-word hyphenated code that is neither real code, POSTed three times.
|
||||
|
||||
| attempt | HTTP | elapsed | message |
|
||||
|---|---|---|---|
|
||||
| 1 | 200 | **1.194 s** | M4 |
|
||||
| 2 | 200 | **1.004 s** | M4 |
|
||||
| 3 | 200 | **1.014 s** | M4 |
|
||||
|
||||
- **Refused each time** ✅ · **no lockout** ✅ (three identical responses; the handler documents
|
||||
`NO LOCKOUT (§8.4)` by design) · **HTTP 200, no error page** ✅
|
||||
- **~1.0–1.2 s is the positive tell that a REAL unseal was attempted** — `age`'s scrypt cost, visible.
|
||||
Phase 1's 0.134 s is what "no attempt was made" looks like. This discriminator is used throughout.
|
||||
- **Nothing was written** ✅ — all four `/data/offbox` files byte-identical with **mtimes unchanged at
|
||||
14:57:35**:
|
||||
|
||||
```
|
||||
846ac6e78a71e868 2026-08-05 14:57:35 applied_marker
|
||||
75dbbc9be6909912 2026-08-05 14:57:35 known_hosts
|
||||
e11a6c542b73477a 2026-08-05 14:57:35 repo_password ← the Phase-3 key, unchanged
|
||||
85579eb0033832a0 2026-08-05 14:57:35 ssh_key
|
||||
```
|
||||
|
||||
- **I4 held** ✅ — a sweep for three fragments of the entered code across the controller's data dir,
|
||||
`/var/log`, `/tmp` and `docker logs` returned **product hits: 0**. **Both controls passed first**:
|
||||
a planted canary was found by the same sweep, and `docker logs … | grep -c recovery` returned 6.
|
||||
The only hits were in the harness's own script (`/tmp/.cc.sh`) — a self-match, cleaned up.
|
||||
|
||||
**The deviation from the brief's expected answer: message 4, not message 1.** F1's stated right
|
||||
answer is *"message 1 only"*. On this box M1 is **unreachable**, because the M4 branch precedes it and
|
||||
fires whenever a superseded package exists.
|
||||
|
||||
Judged on the message rather than the outcome: M4 is *hedged* („**Ha** egy korábbi kódot adtál meg"),
|
||||
it does not accuse the customer of mistyping, and it states two true facts. But a customer who simply
|
||||
**fumbled their current code** is given no hint to re-check their ten words and is instead routed to
|
||||
customer support about *older* backups. **M1 — the only message that tells a customer to check their
|
||||
typing — cannot be reached on any box that has re-escrowed**, which is exactly the population most
|
||||
likely to be typing a recovery code. Recorded as a finding.
|
||||
|
||||
### F2 — the customer has no code at all — **PASS**
|
||||
|
||||
Two halves, both taken.
|
||||
|
||||
- **The screen** states it plainly, unprompted:
|
||||
> „**A helyreállítási kódot senki nem tudja pótolni** — sem a Felhom, sem az ügyfélszolgálat, sem az
|
||||
> üzemeltető. Ez szándékos… Ha a kód elveszett, a korábbi mentések nem nyithatók meg többé."
|
||||
- **It offers the set-aside path rather than a dead end** — „…**nem kérem vissza a korábbi adatokat**"
|
||||
is rendered, because `CanSetAside` (= `OffboxOrphaned()`) is true here. ✅
|
||||
- **An empty submission** returns „**Add meg a helyreállítási kódot.**" in **0.027 s** — no unseal
|
||||
attempted, and no blame attached to a code that was never given. ✅
|
||||
|
||||
### F3 — the hub unreachable when the code is entered — **FAIL**
|
||||
|
||||
Injected on the appliance: `iptables -I OUTPUT 1 -d 37.191.56.193 -j REJECT`.
|
||||
**Control: hub `http=302` before → `http=000, exit=7` after.** The fault is real.
|
||||
|
||||
Then the **correct current recovery code** was entered, so the only possible cause of failure is the hub.
|
||||
|
||||
```
|
||||
HTTP 200 · elapsed 0.0556 s ← far below the ~1.0 s scrypt cost: NO unseal was attempted
|
||||
```
|
||||
|
||||
> **M4** — „Ez a kód nem nyitja meg azt a csomagot, amit most őrzünk ehhez a géphez. Ha egy korábbi
|
||||
> kódot adtál meg: a géped azóta új mentési kulcsot kapott…"
|
||||
|
||||
**The code was perfect and current. The hub was down. The customer is told their code does not open
|
||||
their package and is sent to customer support about older backups.** The hub is never named. **I6
|
||||
breach** — an unreachable service reported as a fact about the code.
|
||||
|
||||
**And the machine knew.** The agent's own journal distinguishes the two cases exactly:
|
||||
|
||||
```
|
||||
F1 (wrong code): err="escrow: the recovery code did not unwrap the identity escrow
|
||||
(wrong recovery code, or a corrupt blob): exit status 1"
|
||||
F3 (hub down): err="escrow: fetching the sealed bundle: hub: transport error:
|
||||
Get \"https://hub.felhom.eu/api/v1/hosts/c11-36d660/escrow\":
|
||||
dial tcp 37.191.56.193:443: connect: no route to host"
|
||||
```
|
||||
|
||||
The discriminator exists in the agent's `err` field and is **discarded at the HTTP boundary**: both
|
||||
return **HTTP 400** with one merged sentence — *"the recovery code did not open the sealed bundle,
|
||||
**or the bundle could not be fetched**"* — and the agent's own `msg=` collapses them too
|
||||
(*"FAILED (wrong recovery code, or the blob could not be fetched)"*). The controller then has no
|
||||
branch for "could not reach", so `rerr != nil` falls into the code/package messages.
|
||||
|
||||
**Reverted**: rule deleted, chain empty, hub `http=302` again.
|
||||
|
||||
### F4 — the agent stopped when the code is entered — **FAIL**
|
||||
|
||||
Injected: `systemctl stop felhom-agent`. **Control: `is-active` = `active` → `inactive`, and no
|
||||
listener on `:8443`.** Real.
|
||||
|
||||
```
|
||||
HTTP 200 · elapsed 0.0299 s ← again no unseal
|
||||
```
|
||||
|
||||
> **M4** — the same sentence as F1 and F3.
|
||||
|
||||
**The stated right answer is M2** — *"names the machine's inability to ask, the R-216 fix under
|
||||
pressure"*. M2 did **not** fire. The controller log names the truth the customer never sees:
|
||||
|
||||
```
|
||||
recovery: unlock failed: agentapi: POST /escrow/recover-offsite-password:
|
||||
Post "https://169.254.253.1:8443/escrow/recover-offsite-password":
|
||||
dial tcp 169.254.253.1:8443: connect: connection refused
|
||||
```
|
||||
|
||||
**Why the R-216 gate did not catch it, measured rather than reasoned** — the box's own debug ring:
|
||||
|
||||
```
|
||||
[web] recovery capability gate: offsite_key_recovery=yes (source=version)
|
||||
```
|
||||
|
||||
**`source=version`.** `recoverySupport` asks `netFeatures.SupportsWithSource(...)`, which answers from
|
||||
the **known agent version** without probe traffic — by design, and correct for the question it was
|
||||
built to answer. But the question it is being used for here is *"can this machine ask right now?"*,
|
||||
and a version number cannot answer that. The gate discriminates **age**, not **reachability**; a dead
|
||||
agent of the right version sails through it, and the failure that follows is attributed to the code —
|
||||
the precise outcome R-216's comment says *"must never"* happen (*"An attempt that cannot succeed must
|
||||
never be made, because its failure is attributed to the code"*).
|
||||
|
||||
**F3 and F4 are one defect with two faces**: on the unlock failure path there is no branch for *the
|
||||
machine could not ask or could not reach*, so every non-code cause is rendered as a statement about
|
||||
the code or the package. This is Phase 1's headline finding, relocated from the version channel to the
|
||||
transport.
|
||||
|
||||
**Reverted**: agent `active`, `:8443` listening, `felhom-agent 0.125.0`.
|
||||
|
||||
### F5 — the store unreachable after a successful unlock — **PASS**
|
||||
|
||||
The first attempt's control **failed and is reported as a failed control**: the probe used `nc`, which
|
||||
the controller container does not have, so blocked and unblocked both printed the same fallback. A
|
||||
second attempt used the IPv6 address `getent hosts` returned — and the container has **no IPv6 route
|
||||
at all**, so that too could not discriminate. The container resolves the store to **IPv4
|
||||
`91.98.242.176`**, and only a `/dev/tcp` probe on that address separated the states:
|
||||
|
||||
```
|
||||
BEFORE: TCP-OPEN INJECT: iptables REJECT (OUTPUT + FORWARD) AFTER: TCP-CLOSED
|
||||
```
|
||||
|
||||
Only then was the fault real. With the **correct current code**:
|
||||
|
||||
```
|
||||
HTTP 200 · elapsed 1.198 s ← a REAL unseal
|
||||
```
|
||||
|
||||
> „**A kulcs visszakerült**, de a mentések listáját most nem sikerült beolvasni. A mentéseid
|
||||
> nincsenek veszélyben — nézd meg a Biztonsági mentés oldalt néhány perc múlva."
|
||||
|
||||
- **The unlock still counts** ✅ · **M3, the "could not be read" shape** ✅
|
||||
- **R-217's false claim is genuinely absent** ✅ — verified in Python on the local file with UTF-8
|
||||
matching, *after* the accented-substring trap was accounted for (the project's own documented false-
|
||||
zero hazard):
|
||||
|
||||
```
|
||||
MUST BE ABSENT "A tároló megnyílt" 0 · "van benne tartalom" 0 · "nem tudtuk alkalmazásokhoz rendelni" 0
|
||||
CONTROLS PRESENT "A kulcs visszakerült" 1 · "nézd meg a Biztonsági mentés oldalt" 1
|
||||
```
|
||||
|
||||
**R-217's fix holds under exactly the fault it was written for.**
|
||||
|
||||
**Reverted**: rules deleted, `TCP-OPEN` again.
|
||||
|
||||
### F6 — „Most nem", then return — **PASS**
|
||||
|
||||
Baseline first (the interception is on `/launcher` and `/dashboard`, not `/`):
|
||||
|
||||
```
|
||||
BEFORE GET / -> 302 /launcher
|
||||
GET /launcher -> 302 /recovery ← the full-page interruption
|
||||
GET /dashboard -> 302 /recovery
|
||||
GET /recovery -> 200
|
||||
```
|
||||
|
||||
`POST /recovery/postpone` → **302 → /launcher**, and:
|
||||
|
||||
```
|
||||
AFTER GET /launcher -> 200 ← no longer interrupted
|
||||
GET /dashboard -> 200
|
||||
GET /recovery -> 200 ← still reachable, permanently
|
||||
```
|
||||
|
||||
**The backups-area entry point SURVIVES** ✅ — `/backups/remote` still carries exactly one `/recovery`
|
||||
link and the sealed-package sentence, before and after:
|
||||
|
||||
> „A korábbi, házon kívüli mentéseid visszaszerezhetők… **Adatok visszaszerzése**"
|
||||
|
||||
That asymmetry — full page silenced, entry point permanent — is what the handler documents, and it
|
||||
holds. **And the unlock still works after postponing** (measured: HTTP 200, 3.214 s, a real unseal).
|
||||
|
||||
|
||||
### F8 — the controller restarted mid-unlock — **state PASS, message FAIL**
|
||||
|
||||
Injected: an unlock started, then `docker restart felhom-controller` at **T+0.7 s**, inside the
|
||||
~1.1 s unseal window. **Control: the container's `StartedAt` moved `2026-08-05T15:57:04Z` →
|
||||
`2026-08-05T21:06:05Z`.** The restart really landed mid-request.
|
||||
|
||||
What the customer got:
|
||||
|
||||
```
|
||||
HTTP 502 · elapsed 1.914 s · body: "Bad Gateway"
|
||||
```
|
||||
|
||||
- **No half-state** ✅ — after the restart the four `/data/offbox` files are byte-identical with
|
||||
**mtimes still 14:57:35**, and `settings.json` differs from the pre-F8 snapshot in exactly one
|
||||
field, `postponed: True`, which F6 set. The controller returned **healthy in 40 s**.
|
||||
- **„the screen says which" — it does not** ❌. The customer is shown **`Bad Gateway`**: a raw
|
||||
upstream error, in **English**, from traefik. It names no reason, offers no action, and says
|
||||
nothing about whether the key was installed. **I3 breach** — *"every refusal names a reason a
|
||||
person can act on, in Hungarian, with no raw error."*
|
||||
|
||||
> **What F8 does NOT establish, stated so the pass is not overclaimed.** On this venue the unlock
|
||||
> resolves to `RecoverUnchanged` (the recovered password is byte-identical to the one already on
|
||||
> disk), so `InjectOffboxPassword` — the only writing branch — **was never reached**. F8 therefore
|
||||
> shows that a mid-request restart leaves state coherent and returns a raw 502; it does **not**
|
||||
> establish that the password *write* is atomic. Testing that needs shape (a), which the venue does
|
||||
> not hold. Recorded as a gap, not as a pass.
|
||||
|
||||
#### §4.1 again — and this time the box says it out loud
|
||||
|
||||
The brief expected the floor positive here, and F8's restart delivered a better one than `/settings`:
|
||||
a **cold-started process** stating the floor it was served, ten seconds in. The same log line, same
|
||||
box, same code, before and after the vouch:
|
||||
|
||||
```
|
||||
15:58:35 [offsite-apply] settle-gate: GO — floor still unknown after 1m30s ← hold IN FORCE
|
||||
...
|
||||
21:06:06 [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK)
|
||||
21:06:16 [offsite-apply] settle-gate: GO — at/above floor 0.200.0 (we are 0.201.0),
|
||||
no managed update running ← hold RELEASED
|
||||
```
|
||||
|
||||
**§4.1 is therefore measured twice, independently**: the rendered `GetFloor()` value, and this
|
||||
box-side line from a fresh process that had no prior in-memory state to inherit.
|
||||
|
||||
---
|
||||
|
||||
## The finding F6 turned up on the way past — the store reports 0 when it cannot read
|
||||
|
||||
Not one of the eleven; found while checking F6's *"the listing is coherent"* clause on
|
||||
`/backups/remote`. It is reported because a suspicion checked and confirmed is a result.
|
||||
|
||||
The page renders, **on one screen**:
|
||||
|
||||
```
|
||||
Tároló méret · 0 pillanatkép Tárhelykeret: 0 / 50 GB (0%)
|
||||
```
|
||||
|
||||
and, immediately below:
|
||||
|
||||
> „**A távoli tároló másik kulccsal készült mentéseket tartalmaz.** A távoli tárhelyen lévő mentések
|
||||
> egy korábbi, már nem elérhető kulccsal készültek… **A meglévő mentések nem sérültek**…"
|
||||
|
||||
**Ground truth, measured directly against the Storage Box over SFTP — a read-only listing, no
|
||||
decryption, using the box's own transport credential:**
|
||||
|
||||
```
|
||||
/home/felhom-repo/snapshots:
|
||||
-rw------- 253 Aug 5 13:14 f3d9cd67d539c00359f0454ea7a782f45beac8406c55275bee5a6886afa8d791
|
||||
/home/felhom-repo: du -s → 12535 (KB)
|
||||
/home/felhom-repo/keys: exactly ONE key
|
||||
```
|
||||
|
||||
That is **snapshot `f3d9cd67`** — the Phase 0 snapshot holding all three sentinels, byte-size and
|
||||
timestamp matching the journal's `repo_size_bytes 12 611 522` and its `13:13:32Z` capture.
|
||||
|
||||
> **One snapshot and ~12.5 MB of the customer's only surviving copy, rendered as `0 pillanatkép`
|
||||
> and `0 / 50 GB`, on the page the customer opens to find out whether their backups still exist —
|
||||
> directly above a paragraph telling them the store contains backups.**
|
||||
|
||||
**Mechanism, from the box's own state rather than inferred.** After the Phase 3 rebuild the
|
||||
controller's `settings.json` `offbox` block carries **no `snapshot_count` and no `repo_size_bytes`
|
||||
key at all** (the full block is quoted in the pre-flight section above). The values are *unknown*,
|
||||
and unknown renders as the zero value.
|
||||
|
||||
This is **R-217's defect class in a second location**: a field whose zero value is indistinguishable
|
||||
from a real measurement, defaulted past on a failure/unknown path. `OffsiteInventory.Empty` exists
|
||||
precisely because *"len(Apps)==0 is also what a failed read looks like"* — the same reasoning applies
|
||||
to a snapshot count the box has never successfully taken. **I6**: an absence reported as a fact.
|
||||
|
||||
**I5 checked and NOT breached** — the hub's `/offsite` page shows `Campaign 11 · 0.0 GB · 50 GB`, but
|
||||
that is 12.5 MB rounded to one decimal of a GB, and the pool total (`Used 3.8 GB`) is consistent. The
|
||||
hub and the box do not disagree; **both** understate, for different reasons. Only the box's
|
||||
**snapshot count** — an integer, not a rounding — is false.
|
||||
|
||||
+28
-8
@@ -1,3 +1,31 @@
|
||||
## v0.97.1 — the held-floor REASON must match the hold's CAUSE (2026-08-05, CAMPAIGN-11 follow-on)
|
||||
|
||||
> **Entry added retroactively 2026-08-05 (CAMPAIGN-11 Phase 2).** The change shipped and **is** the
|
||||
> deployed image (`felhom-hub:0.97.1`, commits `a7f1d27` + `79e31ac`), but it was written into the
|
||||
> **v0.97.0** entry instead of its own — so a reader matching the running tag against this file found
|
||||
> no `v0.97.1` heading at all. Second occurrence of a version deployed without its own entry (the
|
||||
> first was agent `0.90.1`). Nothing about the change is new here; only its heading is.
|
||||
|
||||
Adding a second hold reason and leaving both surfaces printing the first is `CLAUDE.md`'s corollary,
|
||||
and it happened here: the freshly deployed hub logged
|
||||
|
||||
```
|
||||
managed floor HELD for c11: agent "0.125.0" < MinAgent 0.113.0
|
||||
```
|
||||
|
||||
— a comparison that is **FALSE** (0.125.0 is *above* 0.113.0) for a box held because its floor sat
|
||||
above the vouched golden. Both the ACK log line and the Hosts-dashboard `HeldReason` now come from one
|
||||
`ManagedFloorDecision.HoldReason()`, with `RequestedFloor` kept so the message can name the floor that
|
||||
was asked for (`Floor` is cleared to `""` on a hold, and a reason naming an empty floor tells the
|
||||
operator nothing). `TestResolveManagedFloor_HoldReasonMatchesTheCause` pins each reason to its cause.
|
||||
**A true alarm that misdescribes itself is one the operator learns to dismiss.**
|
||||
|
||||
**Confirmed live in CAMPAIGN-11 Phase 2**: the corrected sentence is what the hub logged for c11 every
|
||||
15 minutes until the golden rebake released the hold —
|
||||
*"held: floor 0.200.0 is ABOVE the vouched golden 0.192.0, so its agent requirement is unknown — vouch
|
||||
a golden carrying the floor's controller (publish-train rule 1)"* — and it stopped, as it should, once
|
||||
golden 0.201.0 was vouched.
|
||||
|
||||
## v0.97.0 — the floor stops being served past the agent it depends on (2026-08-05, CAMPAIGN-11)
|
||||
|
||||
**R-216, the hub half.** `ResolveManagedFloor` exists to *"never push a controller past the agent it
|
||||
@@ -41,14 +69,6 @@ either way. **Only rows that actually carry an identity blob count**: the pre-v0
|
||||
retain nothing the screen could be talking about. It is a boolean and a timestamp; it grants **no read
|
||||
path**, which is still unbuilt (R-199's inventory).
|
||||
|
||||
**And the hold REASON now matches the hold CAUSE.** Adding a second reason and leaving both surfaces
|
||||
printing the first is `CLAUDE.md`'s corollary, and it happened here: the freshly deployed hub logged
|
||||
`managed floor HELD for c11: agent "0.125.0" < MinAgent 0.113.0` — a comparison that is FALSE
|
||||
(0.125.0 is above 0.113.0) for a box held because its floor sat above the golden. Both the ACK log
|
||||
line and the Hosts-dashboard `HeldReason` now come from one `ManagedFloorDecision.HoldReason()`, and a
|
||||
test pins each reason to its cause. A true alarm that misdescribes itself is one the operator learns
|
||||
to dismiss.
|
||||
|
||||
Tests: `managed_floor_test.go` — the campaign's exact numbers, held-regardless-of-agent, Scenario C
|
||||
(uncoupled untouched), floor==golden unchanged, unparseable golden. **Red-proof: removing the
|
||||
floor-above-golden branch reproduces the campaign's measurement exactly.**
|
||||
|
||||
Reference in New Issue
Block a user