Compare commits
2 Commits
f88faad50b
...
07998dcff2
| Author | SHA1 | Date | |
|---|---|---|---|
| 07998dcff2 | |||
| ce6fafa14f |
@@ -4,8 +4,8 @@
|
||||
supersession) PASSED on its central question. Phase 2 (eleven injected faults) and Phase 4 (an
|
||||
unattended soak) ran overnight on 2026-08-05/06 and are reported here for the first time.**
|
||||
|
||||
**Findings: R-214 … R-223 from Phases 1 and 3** (seven since fixed and shipped), **plus R-224 … R-227
|
||||
from Phase 2.** **Three suspicions investigated — two confirmed, one DISPROVED.** **Four harness
|
||||
**Findings: R-214 … R-223 from Phases 1 and 3** (seven since fixed and shipped), **plus R-224 … R-228
|
||||
from Phase 2.** **Three suspicions investigated — two confirmed, one DISPROVED.** **Five harness
|
||||
faults, separated from the product's.**
|
||||
|
||||
Evidence: `../tests/campaign11-evidence-2026-08-05/` — `journal.md` (Phases 0, 1, 3) and
|
||||
@@ -155,13 +155,66 @@ was real; **every control that failed is reported as a failed control, not as a
|
||||
| **F4** | agent stopped | **M2**, the R-216 fix under pressure | **FAIL** — M4. → **R-224** |
|
||||
| **F5** | store unreachable after a successful unlock | unlock counts; „could not be read", not „opened with content" | **PASS** — R-217's fix holds |
|
||||
| **F6** | „Most nem", return later | entry point survives; unlock still works | **PASS** |
|
||||
| **F7** | set aside, then a change of mind | set aside, never deleted; honest afterwards | *(see §4.7)* |
|
||||
| **F7** | set aside, then a change of mind | set aside, never deleted; honest afterwards | **SPLIT** — set-aside **PASS**, verified byte-for-byte; the afterwards **FAIL** → **R-228** |
|
||||
| **F8** | controller restarted mid-unlock | no half-state; the screen says which | **PARTIAL** — no half-state ✅, but a raw English `Bad Gateway` → **R-227** |
|
||||
| **F9** | a box that never had off-site | nothing, incl. a direct `GET /recovery` | *(see §4.9)* |
|
||||
| **F10** | app with a mandatory data path missing | run reports `incomplete`, names the app, digest carries it | *(see §4.10)* |
|
||||
| **F11** | box offline a whole reporting window | hub and box agree once it returns | *(see §4.11)* |
|
||||
| **F9** | a box that never had off-site | nothing, incl. a direct `GET /recovery` | **PARTIAL PASS** — R-215's gate proven live on a narrower shape; the literal precondition was not staged |
|
||||
| **F10** | app with a mandatory data path missing | run reports `incomplete`, names the app, digest carries it | **NOT INJECTED** — harness. Three attempts, each self-healed |
|
||||
| **F11** | box offline a whole reporting window | hub and box agree once it returns | **PASS** — both directions fired, each with an operator mail |
|
||||
|
||||
*(F7, F9, F10 and F11 are written up in the sections below; the table is the index.)*
|
||||
### F7 — the set-aside, and the afterwards
|
||||
|
||||
**The move-aside is exactly what it claims.** Both confirmations state the consequences first
|
||||
(*„félretesszük — nem töröljük"*, *„a gép új, üres mentési tárolót kezd"*, *„ez az oldal többé nem
|
||||
jelenik meg"*), and the result measured against the Storage Box:
|
||||
|
||||
```
|
||||
BEFORE /home/felhom-repo mtime 13:13 snapshot f3d9cd67 du -s 12535
|
||||
AFTER /home/felhom-repo.orphaned-20260805 mtime 13:13 snapshot f3d9cd67 du -s 12535 ← untouched
|
||||
/home/felhom-repo mtime 21:32 (empty) ← fresh
|
||||
```
|
||||
|
||||
**Nothing deleted.** Then the customer changes their mind — and finds nothing: `GET /recovery` 302s,
|
||||
`POST /recovery/unlock` 302s in 0.028 s with no message, and `/backups/remote` never mentions the
|
||||
set-aside history at all, though the box records its exact path. → **R-228**
|
||||
|
||||
### F9 — what was and was not proven
|
||||
|
||||
The literal precondition (a box that never had off-site backups) needs a rebuild, which the brief
|
||||
forbids before Phase 4, so it was **not staged**. But **the assertion that failed in Phase 1 was
|
||||
tested and passes**: Phase 1's defect was that `GET /recovery` never asked the predicate. After F7
|
||||
drives `recoveryOffer()` false, `GET /recovery` → **302 `/backups/remote`**. **R-215's fix is proven
|
||||
live.** The `GetHubEscrowIdentityPresent()==false` arm remains covered by tests only.
|
||||
|
||||
### F10 — not injected, and why that is a harness result
|
||||
|
||||
Three attempts, each with a control confirming the directory was genuinely absent — and each
|
||||
self-healed before capture: (1) the app recreated it; (2) `docker stop` was undone by **the
|
||||
controller's own monitor restarting the stack**; (3) with the stack stopped through the controller
|
||||
API, the directory reappeared at **21:51:38.65**, coincident with the run's own start. All three runs
|
||||
reported `ok` with `1 mandatory path(s)`, and R-203's stat-gap correctly never fired, because by the
|
||||
time `os.Stat` ran the path existed.
|
||||
|
||||
**The state F10 describes is not reachable on a deployed app of this kind.** Recorded as **harness,
|
||||
not product**.
|
||||
|
||||
**One observation kept, with its evidence:** at capture the directory held only a recreated
|
||||
`metadata.db` and **not** the customer's `F10-SENTINEL.txt`, and the run still said `ok`. The verdict
|
||||
is about a path's *presence*, not its *content* — correct as designed, and it means an `ok` off-site
|
||||
run can immediately follow the loss of everything that path contained.
|
||||
|
||||
### F11 — the dead-man's switch, both ways
|
||||
|
||||
```
|
||||
23:28:11 ok → stale (host_stale) + Operator email SENT
|
||||
23:29:59 Received report from c11 ← the box returns unaided
|
||||
23:30:11 stale → ok (host_recovered) + Operator email SENT
|
||||
23:30:11 customer mail skipped — no unanswered customer down mail (pairing miss)
|
||||
```
|
||||
|
||||
The customer mail was correctly **withheld** by the pairing gate under a real outage. **I5 holds** —
|
||||
box and hub agree after the return, judged after a full report cycle rather than from one read.
|
||||
**Not reached:** `STALE → DOWN` (>1 h); the outage was ended once both transitions had fired because
|
||||
Phase 4 needed the venue back.
|
||||
|
||||
### The timing discriminator used throughout
|
||||
|
||||
@@ -297,6 +350,32 @@ nothing about whether the key was installed. **I3** — *"every refusal names a
|
||||
on, in Hungarian, with no raw error."* Low severity (the window is ~1 s wide) and recorded rather than
|
||||
inflated.
|
||||
|
||||
### R-228 — the set-aside history becomes invisible the moment it is set aside
|
||||
|
||||
The move-aside is correct and was verified byte-for-byte (§F7). What follows it is not.
|
||||
|
||||
```json
|
||||
"orphaned_renamed_to": "/home/felhom-repo.orphaned-20260805"
|
||||
```
|
||||
|
||||
The box records the exact path. And:
|
||||
|
||||
```
|
||||
grep -rn "OrphanedRenamedTo" internal/web/templates/ internal/web/*.go → (no hits)
|
||||
```
|
||||
|
||||
The field is **written and read by nobody**. `/backups/remote` after the set-aside contains no
|
||||
occurrence of the path, „félretéve", „régi előzmény" or any equivalent — with instrument controls
|
||||
passing (`felhom-repo` → 2, „letétbe helyezve" → 1), so the page and the matcher both work.
|
||||
|
||||
> **12.5 MB of the customer's deliberately retained data sits at a path the box knows and never
|
||||
> shows.** Its only mention is a flash message on the redirect, gone on the next click.
|
||||
|
||||
**The project's own "seam built but never wired" pattern** — the fifth recorded instance — landing on
|
||||
the one promise the set-aside screen makes. The fix is a plain statement that an earlier history is
|
||||
set aside and not deleted; **it must not promise the history can be reopened**, because R-222 means it
|
||||
cannot be, and that is exactly the conditional promise R-202's gate exists to prevent.
|
||||
|
||||
---
|
||||
|
||||
## 6. What behaved correctly
|
||||
|
||||
@@ -60,8 +60,8 @@ drill needed — the fourth is R-193, and until it goes the recovery is still op
|
||||
**The whole journey was walked end to end for the first time, on a throwaway appliance built from the
|
||||
published ISO. The data came back byte-identical; the journey did not exist.** **Ten findings from
|
||||
Phases 1 and 3, R-214 … R-223** (seven fixed in controller **v0.201.0** + hub **v0.97.0/0.97.1**;
|
||||
**three deliberately still open**, each blocking a real flow), **plus four from Phase 2's injected
|
||||
faults, R-224 … R-227.** Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` (Phases 0/1/3)
|
||||
**three deliberately still open**, each blocking a real flow), **plus five from Phase 2's injected
|
||||
faults, R-224 … R-228.** Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` (Phases 0/1/3)
|
||||
and `journal-phase24.md` (Phases 2/4). Campaign document:
|
||||
`audits/CAMPAIGN-11-recovery-journey-2026-08-05.md`.
|
||||
|
||||
@@ -93,6 +93,7 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour
|
||||
| **R-224** | **Every non-code failure on the unlock path is reported to the customer as a statement about their code.** Two faults, one defect. **F3** — the hub REJECTed (control: `302` → `exit 7`) and the **correct current** code returned M4 (*„Ez a kód nem nyitja meg azt a csomagot…"*) in **0.0556 s**. **F4** — `felhom-agent` stopped (control: `:8443` gone) and the same correct code returned M4 in **0.0299 s**. Neither elapsed time is compatible with a real unseal (~1.0 s of scrypt; F1's wrong-code attempts took 1.194/1.004/1.014 s), so **no unseal was attempted in either case** — the failure was upstream and was attributed to the code anyway. **The discriminator exists and is thrown away**: the agent's own `err` reads `fetching the sealed bundle: hub: transport error: … no route to host` (F3) versus `the recovery code did not unwrap the identity escrow` (F1), but both return **HTTP 400** under one merged sentence (*"…or the bundle could not be fetched"*), and the controller's failure path has **no branch for "could not ask / could not reach"**. **⚠ The vocabulary already exists and is wired to the wrong condition** — „A gép házon belüli kapcsolata most nem elérhető" fires only when `s.agentClient()` fails to *construct*, never when the RPC fails. **And R-216's gate cannot catch it**: the box's own ring reads `recovery capability gate: offsite_key_recovery=yes (source=version)` — the gate discriminates the agent's **age**, not its **reachability**, so a dead agent of the right version sails through the guard whose own comment says *"An attempt that cannot succeed must never be made, because its failure is attributed to the code."* **This is Phase 1's headline finding relocated from the version channel to the transport.** Breaches **I6** | **OPEN — NOT FIXED.** Fix shape: classify `rerr` (transport/fetch vs unseal) at the agent's HTTP boundary and give the controller a reach-failure branch. Do NOT widen the capability gate into a liveness probe on the unlock path — a probe that must succeed before an unlock adds a second thing that can fail |
|
||||
| **R-225** | **The remote store reports `0 pillanatkép · 0 / 50 GB` when the box cannot read it — directly above a card stating the store holds backups.** `/backups/remote` renders both at once. **Ground truth, measured over SFTP with the box's own credential (read-only, no decryption):** `/home/felhom-repo/snapshots` holds **`f3d9cd67…`** — the Phase 0 snapshot with all three sentinels, the customer's only surviving copy — and `du -s` returns **12 535 KB**, matching the journal's `repo_size_bytes 12 611 522`. **Mechanism, from the box's own state:** after the rebuild the `offbox` block carries **no `snapshot_count` and no `repo_size_bytes` key at all** — the values are *unknown* and unknown renders as the zero value. **This is R-217's defect class in a second location**, and `OffsiteInventory.Empty` exists precisely because *"len(Apps)==0 is also what a failed read looks like"*. **I5 checked and NOT breached** — the hub's `0.0 GB` is 12.5 MB rounded, and the pool total is consistent; only the box's snapshot **count**, an integer, is false. Breaches **I6** | **OPEN — NOT FIXED.** A count the box has never successfully taken must render as unknown („—"), never as `0` |
|
||||
| **R-226** | **M1 — the only message that tells a customer to check their typing — is unreachable on any box that has re-escrowed.** The failure path tests M4's condition *before* M1's (`if present, at := s.recoverySuperseded(); present { …M4…; return }`), so on every box the hub keeps an earlier package for, a **genuinely mistyped code** produces M4. **Measured**: F1's three wrong-code attempts each returned M4 after a real ~1 s unseal. M4 is hedged (*„**Ha** egy korábbi kódot adtál meg"*) and states two true facts, but it offers no hint to re-check the ten words and routes the customer to support about *older* backups. **The population that has re-escrowed is exactly the population that has just been handed a new recovery code and is most likely to be typing one.** R-222's fix removed one conflation and introduced another on the same branch | **OPEN — NOT FIXED.** Fix shape: M4 should name both possibilities (a mistype OR an earlier code), since from the engine's side they genuinely are indistinguishable — which is the honest thing to say |
|
||||
| **R-228** | **After „I do not want the old data", the set-aside history becomes invisible — the box records where it is and shows it to nobody.** The move-aside itself is **correct and was verified byte-for-byte**: `/home/felhom-repo` → `/home/felhom-repo.orphaned-20260805` with its mtime, its `du -s` (**12 535 KB**) and snapshot **`f3d9cd67`** all unchanged, and a fresh empty repo initialised beside it. **Nothing was deleted.** But `settings.json` then carries `"orphaned_renamed_to": "/home/felhom-repo.orphaned-20260805"` and a census returns **zero** references to `OrphanedRenamedTo` in any template or web handler — the field is written and read by nobody. `/backups/remote` after the set-aside contains no occurrence of the path, „félretéve", „régi előzmény" or any equivalent (instrument controls: `felhom-repo` → 2, „letétbe helyezve" → 1). **12.5 MB of the customer's deliberately retained data sits at a path the box knows and never shows**; its only mention is a flash message on the redirect, gone on the next click. Meanwhile `GET /recovery` → 302 and `POST /recovery/unlock` → 302 with no message, so a customer who changes their mind gets **no explanation at all** (correctly, not a typing accusation — but not an explanation either). **The project's own "seam built but never wired" pattern**, landing on the one promise the set-aside screen makes | **OPEN — NOT FIXED.** Fix shape: surface `OrphanedRenamedTo` on `/backups/remote` as a plain statement that an earlier history is set aside and not deleted. **Do NOT promise it can be reopened** — R-222 means it cannot be, and R-202's gate is precisely about not making that conditional promise |
|
||||
| **R-227** | **A controller restart mid-unlock returns a raw English `Bad Gateway`.** F8 restarted the container at T+0.7 s, inside the unseal window (control: `StartedAt` moved). The customer got **HTTP 502 / „Bad Gateway"** from traefik — a raw upstream error, in English, naming no reason and saying nothing about whether the key was installed. **The state half is clean**: the four `/data/offbox` files stayed byte-identical with mtimes frozen, and the controller returned healthy in 40 s. Breaches **I3** | **OPEN — NOT FIXED.** Low severity — the window is ~1 s wide. Recorded rather than inflated |
|
||||
|
||||
**Recorded against existing rows by Phase 2:**
|
||||
|
||||
@@ -641,3 +641,67 @@ its edge.
|
||||
| Recovery screen | retired (correct — the set-aside declared it would be) |
|
||||
| Deviations left in place | the raw `/mnt/adatok` and `/mnt/mentes` mounts remain **unmounted** (R-220's workaround); the stable `/mnt/felhom-drives/*` mounts are what everything uses |
|
||||
|
||||
|
||||
---
|
||||
|
||||
# Phase 4 — leave it alone and watch
|
||||
|
||||
**Window: 2026-08-05 23:56 CEST → 2026-08-06 ~04:45 CEST.** The venue is not touched: no endpoint is
|
||||
driven, no service restarted, nothing configured. Only logs and the hub are read.
|
||||
|
||||
## Pre-registered expectations — written BEFORE the window, so the comparison is not fitted afterwards
|
||||
|
||||
**A soak that reports only "nothing broke" has measured nothing.** Both directions are named here in
|
||||
advance. Sources: the box's own `[scheduler] Registered …` lines and the agent's own `backup tier
|
||||
armed` lines, read at 23:56.
|
||||
|
||||
### The box's own schedule (controller, restarted 23:29:53 CEST after F11)
|
||||
|
||||
| when | job |
|
||||
|---|---|
|
||||
| **02:30** | `db-dump` |
|
||||
| **03:30** | `tier2-backup` · `fill-watch` |
|
||||
| **04:00** | `metrics-prune` |
|
||||
| **04:15** | `offbox-backup` ← **the one this campaign cares about** |
|
||||
| every 15 m | `hub-report` |
|
||||
| every 1 m / 5 m / 2 m | `agent-channel-health` · `system-health` · `stack-scan` |
|
||||
| every 10 s / 30 s | `status-refresh`, `health-probes` · `deadapp-check`, `ring-spill` |
|
||||
| every 6 h | `selfupdate-check` · `disk-health-check` |
|
||||
|
||||
### The agent's schedule (host domain)
|
||||
|
||||
```
|
||||
backup tier armed target=felhom-backup cadence=24h keep_last=3 primary=true
|
||||
backup tier armed target=felhom-pbs cadence=168h keep_last=0 primary=false
|
||||
backup: restore-test scheduler starting (per-archive due-check) eval_interval=6h settle=24h
|
||||
```
|
||||
|
||||
### MUST happen (an absence here is a finding)
|
||||
|
||||
1. Reports keep arriving every ~15 min; the host stays **ONLINE** all night.
|
||||
2. **`db-dump` at 02:30**, **`tier2-backup` + `fill-watch` at 03:30**, **`metrics-prune` at 04:00**.
|
||||
3. **`offbox-backup` at 04:15** runs unprompted and succeeds — `snapshot_count` **1 → 2**,
|
||||
`last_status: ok`, `last_success` stamped in the 04:15 window.
|
||||
4. A **`backup_run_digest`** event reaches the hub after a backup run (this is the "operator digest";
|
||||
it is event-driven per run, **not** a nightly cron — checked in the hub source rather than assumed).
|
||||
5. The agent's **`felhom-backup`** tier (24 h cadence) takes a whole-guest archive when it comes due.
|
||||
|
||||
### MUST NOT happen (firing here is equally a finding)
|
||||
|
||||
1. **No `host_stale` / `host_recovered`** — the box is up and staying up.
|
||||
2. **No `offsite_selfheal_restaged`** — the box has a target, so `needsOffsiteCredential` returns
|
||||
false and it must not declare. *(This is also §4.2's negative control: if the box DOES declare
|
||||
tonight with a target present, R-218's fix is wrong in the other direction.)*
|
||||
3. **No `offbox_repo_orphaned`** — the repo is fresh and written under the key the box holds.
|
||||
4. **No `offsite_repo_key_changed`** — nothing re-escrows tonight.
|
||||
5. **No `escrow blob SERVED`** — nobody is driving a recovery.
|
||||
6. **No controller self-update** — `auto-update: false`, and the box is at 0.201.0 with floor 0.200.0.
|
||||
|
||||
### Genuinely uncertain, recorded as open questions rather than predictions
|
||||
|
||||
- **The restore-test** evaluates every 6 h with a **24 h settle**, and every archive on this box is
|
||||
hours old. **Expected NOT to run** — and if it does not, that is correct behaviour, not a gap. The
|
||||
distinction matters because "no restore-test fired" would otherwise read as a failure.
|
||||
- Whether the agent's local tier fires tonight depends on when its 24 h cadence last elapsed on a box
|
||||
built today.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user