admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

drill R-201: prepared and HALTED before the wipe — a mandatory data directory was missing from the off-site snapshot while the run said ok (R-203)
gates / gates (push) Successful in 8s
Details

Browse Source
This commit is contained in:
authentik Default Admin
2026-08-04 15:00:06 +02:00
parent 846253baa8
commit a0c4b607a6
7 changed files with 457 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+21
View File
@@ -17,6 +17,27 @@
## Standing rulings
**S-30 — the R-201 drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04). Nothing was wiped.**
It stopped at step 4 because the sentinel file was **not in the off-site snapshot** while the run
reported `ok` (**R-203**): the deploy-time `${USERDATA_PATH}` root and the backup-time `NamespaceRoot`
disagree for an app whose `HDD_PATH` equals `system_data_path` — and the same compose used both roots.
Wiping would have destroyed the only copy of the sentinel and proven nothing.
*Established live on the way, all firsts, and all worth carrying:*
- **A rebuilt box's off-site run REFUSES** — `repo_state: orphaned`, the orphan card, and
`offbox_repo_orphaned` at the hub. It does **not** silently start a fresh history. Closes R-193's Q3.
- **The orphan reset works**: move-aside to `/home/felhom-repo.orphaned-20260804`, never delete.
- **demo-hp's pre-rebuild off-site history is permanently unrecoverable** — its key is in superseded
row id 3 with `identity_blob` NULL, superseded **four hours before** v0.93.0 fixed the retention.
- **Neither pre-existing off-site-toggled app has a restorable file leg** — both are named-volume-only,
which the tier tars but the customer restore never unpacks. Any drill needing a recoverable file
needs an app with a **mandatory userdata path**; `calibre-web` is now deployed on demo-hp as that
fixture, with the sentinel in place.
*Still not established, unchanged:* **no file has ever been restored from an off-site backup after a
wipe**, and Part 0's install path (controller v0.196.0) has never run against a live recovery.
**S-29 — a box may fetch its OWN sealed recovery blob with its OWN credential; the operator-driven DR
path is a separate thing and stays gated (2026-08-04, R-199; hub v0.94.0 + agent v0.125.0 + controller
v0.195.0).**
REPORT.md
+130 -183
View File
@@ -1,221 +1,168 @@
# REPORT — the offsite backup key comes back (2026-08-04, R-199 closed, R-200 half)
# REPORT — R-201 drill: PREPARED, HALTED BEFORE THE WIPE (2026-08-04)
**Class:** implementation across three repos + one decisive live proof. Nothing wiped, deleted,
rotated or reconfigured; no customer-facing change in any language.
**Nothing was wiped. Nothing irreversible happened.** The drill stopped at step 4 because the sentinel
file was not in the off-site snapshot — wiping would have destroyed the only copy and proven nothing.
---
## 1. SCENARIO A — THE RESULT, FIRST, BECAUSE IT IS THE SESSION
## 1. THE VERDICT — not reached, and that is the correct outcome
**On `demo-felhom`, 2026-08-04 13:49 CEST, through the real endpoints end to end:**
The pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established the sentinel
is not in the off-site snapshot at all**, so there was nothing to recover and the verdict is
unobtainable. The runbook's own rule applies: *a precondition was wrong, and the finding outranks the
drill.*
**Sentinel (step 3), still on the box:**
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt`, 181 B.
## 2. Snapshot count at step 9 — N/A; at step 4 it was 3, and the sentinel was in none of them
```
=== offsite key recovery check (R-200) — compares, never installs ===
on-disk sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
recovered sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
[MATCH] the offsite repository password IS recoverable from the sealed escrow.
Nothing was written: this check compares and never installs.
EXIT=0
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
snapshot_count 3 · repo_size_bytes 30 636
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
/mnt/sys_drive/felhom-data/userdata/media/books
[INFO] [offbox] backed up calibre-web (…, 0 mandatory path(s))
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
```
**The hashes matched.** And they match a **third, independent** source: `host_escrow.restic_pw_sha256`
on the hub reads `c60c8bc737a6…` for `demo-felhom-8363b5`, recorded at the ceremony and never involved
in this comparison. The box's file, the sealed bundle and the hub's record all agree.
**The run reported `ok`.** → **R-203**, §9.
**The offsite repository password is recoverable from the sealed escrow with the customer's recovery
code.** That had never been demonstrated — the one live round-trip on record (2026-06-10) predates the
field by a month.
## 3. Preconditions P1–P6
## 2. Part 0 — the recovery code
| # | Result |
|---|---|
| **P1** recovery code held | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file. The code appears nowhere in this report and **was never used** — no unseal ran on demo-hp. |
| **P2** versions | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed published v0.125.0 (sha `f7d8339b53d9…`, verified against the release output) + controller v0.196.0. `age` present at `/usr/bin/age`. |
| **P3** the blob seals the repo under test | **PASS with the caveat that reshaped the drill** — `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`; but **no repository existed under that key** (§5). |
| **P4** verified rollback archive | **NOT TAKEN, deliberately** — only needed for the wipe, and the wipe did not happen. |
| **P5** demo-felhom untouched | **PASS** — not touched at any point. |
| **P6** space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 %, guest 64 GB free. |
**Answered by the operator before the session began:** the 2026-08-04 07:16/07:20 codes had not been
kept, so **the operator ran a fresh ceremony on both nodes and saved the codes** (supplied to this
session as `R_DEMO-FELHOM` in the DooPlex credentials file). No ceremony was run by CC.
## 4. Every step's observable
Measured hub-side, and worth recording because it exercised two of yesterday's behaviours live:
the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z` (demo-felhom), and
`restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated them as
same-password re-ceremonies: **no superseded row was created** and **no `offsite_repo_key_changed`
fired**. That is v0.93.0's Scenario E, live. It also means yesterday's `identity_blob` retention is
still unit-proven and structurally verified, **not yet exercised live** — no supersession has happened.
| step | observable |
|---|---|
| 1 | `identity_blob` 572 B; `restic_pw_sha256` `8a9e33aa4da6…`; `snapshot_count: 0`, **no `last_run`/`last_status` at all** — never ran an off-site backup in this lifetime |
| 1b | **repository ORPHANED** — see §5 |
| 1c | **reset** (operator-authorised) — move-aside to `/home/felhom-repo.orphaned-20260804`, fresh repo initialised, `offbox_repo_reset` pushed |
| 2 | recovery codes for both boxes created and saved by the operator earlier the same day; escrow rows stamped `11:11:37Z` / `11:13:06Z` with **`restic_pw_sha256` unchanged** on both — correctly treated as same-password re-ceremonies (**no superseded row, no `offsite_repo_key_changed`**): v0.93.0's Scenario E, live |
| 3 | `calibre-web` deployed through the real API (HTTP 202), toggled off-site, Tier-1 recovery unit captured; sentinel written + `sync`ed + hashed |
| 4 | `ok`, 3 snapshots, 30 636 B — **and the mandatory directory missing** |
| 5–11 | **NOT RUN.** The §7 STOP was never reached: the drill failed its own precondition first |
Subject box: **demo-felhom** (Tier 0, reachable with a baked key, current escrow). Not demo-hp
(break-glass only), not DooPlex, not ep0.
## 5. Step 1b — the first live proof of a prediction, and it is the good news
## 3. The §8.2 decision as built, and where the gate is
The 2026-08-04 spike predicted from source, and could not measure, that a rebuilt box's next off-site
run would hit a **third** outcome — neither reattaching the old snapshots nor silently starting fresh,
but **refusing**. Measured two ways:
**Built as §8.2:** `GET /api/v1/hosts/{host_id}/escrow`, authenticated with the box's own per-host key,
self-scoped (global may read any — the same asymmetry the PUT has).
**Read-only probe** (`restic cat config` with the current key, writes nothing):
`Fatal: wrong password or no key found` — the exact string `classifyResticProbe` maps to `orphaned`.
**The single gate point is `api.escrowSelfServiceRetrieval`** (`hub/internal/api/handler.go`), a named
constant. `false` re-imposes the recovery-mode requirement on this endpoint and changes nothing else —
the same unseal, the same extraction, the same proof, minus the self-service property. The trade is
written above the handler in `dr.go`'s style: what it gives out, why it is safe, and **what it changes
about who is required** — obtaining the blob used to need an operator to arm recovery mode and now
needs only that box's credential. They still cannot open it. The mitigation is the audit row.
**Then the real customer path** (`POST /backup/offbox/run`):
## 4. Files, commits, deployed versions
```
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
no-longer-available key; runs will skip until reset
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
[INFO] Event pushed: offbox_repo_orphaned (warning) …
```
| Repo | Commits | Version | Deployed |
|---|---|---|---|
| `felhom.eu` | `435f4a5` (code) · `dd08926` (manifest) · docs commit below | hub **v0.94.0** | ArgoCD Synced/Healthy, `felhom-hub:0.94.0` |
| `felhom-agent` | `6d79047` | **v0.125.0** | published sha `f7d8339b53d9…`, **verified by independent download**, installed on demo-felhom (`felhom-agent --version` → 0.125.0, `active`). **NOT vouched** |
| `felhom-controller` | `9640e51` | **v0.195.0** | `felhom-controller:0.195.0` on guest 9201, `Up (healthy)` |
`repo_state: orphaned`, orphan card rendered, event at the hub (HTTP 200).
**The system stopped and said so. It did not quietly start a new history.** Closes R-193's Q3.
**Hub:** `internal/api/handler.go` (route + `handleHostEscrowGet` + `escrowSelfServiceRetrieval`),
`internal/notify/dispatcher.go` (`escrow_blob_served` → `operatorOnlyEvents`),
`internal/api/escrow_get_test.go` (new).
**Agent:** `internal/escrow/recover.go` (new — `OffsiteKeyRecoverer`, distinct errors),
`internal/hub/client.go` (`FetchIdentityEscrow`), `internal/localapi/escrow_recover.go` (new route),
`internal/localapi/server.go` (seam + registration), `cmd/felhom-agent/main.go` (production wiring +
the §8.6 selftest message fix), plus `recover_test.go` and `escrow_recover_wiring_test.go` (new).
**Controller:** `internal/backup/offbox_recovery_check.go` + `offbox_recovery_cli.go` (new),
`internal/agentapi/escrow.go` (`RecoverOffsiteRepoPassword`), `cmd/controller/main.go`
(`--recover-offsite-check`), plus `offbox_recovery_check_test.go` (new).
**Why it was orphaned:** the 15 snapshots / 40.9 MB were written under key `8e03eddf9ff7…` before the
2026-08-03 rebuild. That key survives only in superseded escrow row **id 3**, whose `identity_blob` is
**NULL** — superseded at `2026-08-04 07:15:36`, **four hours before hub v0.93.0 fixed the retention**.
Permanently unrecoverable, with or without a recovery code.
**Green gate in each repo after each phase** (`go build && go vet && go test ./...`, rc=0), plus
`repo_gates.py` / `agent_gates.py` / `controller_gates.py` — all OK. **No test run was combined with a
commit.**
## 6. Step 7's assertion (`identity_blob` unchanged across the rebuild) — NOT REACHED
## 5. Tests and every red-proof
There was no rebuild. Unmeasured, and stated as such.
| Scenario | Result | Red-proof — mutation → outcome |
## 7. Did the built chain carry steps 8–9? — NOT REACHED
Neither the check nor the install ran against a live recovery on demo-hp. The manual substitute was not
needed and was not used. **Part 0 remains unit-proven only.**
## 8. Part 0 — files, commit, tests, deployment
**Commit `1b1366b`**, controller **v0.196.0**, deployed to demo-hp guest 9201 (`Up (healthy)`).
`internal/backup/offbox_recovery_cli.go` (`RecoverAndInstall`), `cmd/controller/main.go`
(`--recover-offsite-install` + `--confirm-install`), `internal/backup/offbox_recovery_check_test.go`.
`go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK. Test run and commit were
never combined.
**Red-proof observed:** removing the confirmation gate makes the dry run write the password —
`TestRecoverAndInstall_InstallsOnABareBox` fails with *"the DRY RUN wrote the password"*.
## 9. R-203 — the finding that halted the drill
| what | path | exists? |
|---|---|---|
| **A** recovered key == on-disk key | PASS (unit, real `age`) + **PROVEN LIVE** | return `bundle.PBSToken` instead of `ResticRepoPassword` → **FAIL**: *"the recovered key is not the sealed repository password (len 24 vs 64)"*. This is the mutation that mattered: every field of that bundle is a non-empty string that looks like a secret |
| **B** wrong R fails closed, nothing written | PASS + **PROVEN LIVE** | — (the property is age's KDF, asserted rather than guarded, so there is no guard to remove) |
| **C** a box fetches only its own blob | PASS | delete the `!isGlobal && authHostID != pathHostID` check → **FAIL**: host A was served host B's blob, body and all |
| **D** no blob → a clean "none" | PASS | — |
| **E** every retrieval leaves a record | PASS + **PROVEN LIVE** | `if false &&` on the audit block → **FAIL**: *"a sealed recovery blob was served and NOTHING recorded it"* |
| **F** R persists nowhere | PASS + **PROVEN LIVE** | write R to a temp file inside the unseal → **FAIL** (see the correction below) |
| **G** the operator DR path is untouched | PASS | — |
| **H** the seam is wired (AST) | PASS | `_ = escrowRecoverer` with the `Options` field removed → **FAIL**: *"localapi.Options … carries no EscrowRecovery field"* |
| §8.5 compare-never-install | PASS + **PROVEN LIVE** | add `InjectOffboxPassword` to the check → **FAIL**: *"the check INSTALLED the recovered password"* |
| the app's live bind (where the sentinel is) | `/mnt/sys_drive/userdata/media/books` | **YES** |
| what the off-site capture set looked for | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
**TWO MUTATIONS DID NOT REMOVE EVERY GUARD ON THE FIRST ATTEMPT, AND BOTH ARE RECORDED RATHER THAN
QUIETLY REDONE** (§9.13 — two of the last four sessions found a red-proof that passed):
**Mechanism, from source:** `NamespaceRoot` (`appbackup/paths.go:28-33`) appends `felhom-data` **when
the drive IS the system data path** (`backup/backup.go:331`); the deploy-time `${USERDATA_PATH}` is
`<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an
app at `HDD_PATH=/mnt/sys_drive`, they differ. **The same compose used both roots** —
`${IMPORT_PATH}` → `…/felhom-data/userdata/import/calibre`, `${USERDATA_PATH}` →
`/mnt/sys_drive/userdata/media/books`.
1. **Scenario F.** The planted leak wrote R into the walked directory and **the test still passed** —
because a later failing call OVERWROTE the leak file with the wrong code, and the test only scanned
for the first one. Instrumenting the mutation (printing where it wrote) is what found it. **The test
was wrong, not the mutation.** It now asserts **emptiness** of the TMPDIR tree as its primary check —
nothing under it is created by the test, so any survivor is a leak regardless of content — with the
content scan kept as defence in depth over all three secrets. Re-run: **FAIL**, *"the unseal left
1 file(s) behind under TMPDIR"*.
2. **Scenario H.** Commenting the wiring line out failed to **compile** (`declared and not used`), which
is not a red-proof. The mutation that compiles keeps the variable alive (`_ = escrowRecoverer`) and
drops the struct field. Re-run: **FAIL** as quoted above.
**Measured vs not, because it changes the fix.** MEASURED: the disagreement, the absence from the
snapshot, the `ok` status, and that the only signal is a container-log WARN. **NOT ESTABLISHED:**
whether `HDD_PATH=/mnt/sys_drive` is supported — it was chosen because demo-hp's only registered drive
(`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working), while
`/mnt/sys_drive` was **accepted** (202). **Either branch is a defect** and the row says so.
**Every `-run` filter was verified to have matched** (`-v`, `=== RUN` lines counted) before any result
was read as a proof.
**Not a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths, so
they are unaffected and their snapshots are real.
## 6. The live validation trail, in order
## 10. Teardown — three layers, nothing torn down that proves anything
1. **Wrong code first, deliberately** (13:44:44) — hub: `escrow blob SERVED to host demo-felhom-8363b5
(572 opaque bytes, self_scope=true) — recovery path in use`; agent: `offsite key recovery FAILED …
err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
blob): exit status 1"`; CLI: `[FAIL] … nothing was written.`, exit 1.
**This is Scenario B live AND it proves links 6 and 7 ran independently of the success** — the blob
was fetched and the unseal was attempted and refused.
2. **The real code** (13:49:47) — §1's output, exit 0.
3. **Nothing was written** — `offbox/repo_password` mtime on the live box is still
`2026-08-03 07:18:02`, unchanged by a check that ran at `11:49` UTC. The whole `offbox/` directory is
byte-for-byte the pre-check listing.
| layer | state |
|---|---|
| the guest | `calibre-web` deployed, running, off-site-toggled, sentinel in place. **Kept** — it is the only app on either demo box with a restorable file leg, and the resumed drill needs it |
| the off-site store | fresh, working, 3 snapshots, 30 636 B. Old 15-snapshot history **set aside, not deleted**, at `/home/felhom-repo.orphaned-20260804` (≈41 MB) |
| the host | agent v0.125.0, controller v0.196.0; `pvesm` unchanged beyond normal usage |
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` recorded for `demo-hp`. **No scratch customer records were created — nothing was reinstalled** |
## 7. The retrieval record as it appears at the hub
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — ruled, still owed, and §8.3 forbids
riding it along with a drill.
```
customer_id severity created_at details_json
demo-felhom warning 2026-08-04 11:44:44 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
demo-felhom warning 2026-08-04 11:49:47 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
```
## 11. The capability-map row as it now reads
And the routing, which is the half that shows the register works:
The R-199 row (the KEY is recoverable, PROVEN-LIVE) keeps its scope and gains the back-pointer the
previous update omitted, plus: the attempt to prove the DATA half was **halted before the wipe**, and
**no file has still ever been restored from an off-site backup after a wipe**. The install half
(v0.196.0) is likewise unit-proven only.
```
customer_id channel status created_at
demo-felhom operator sent 2026-08-04 11:44:44
demo-felhom customer skipped 2026-08-04 11:44:44 ← operator_only
demo-felhom operator suppressed 2026-08-04 11:49:47 ← 1h operator cooldown, itself recorded (R-182)
demo-felhom customer skipped 2026-08-04 11:49:47
```
## 12. Registers
**Both retrievals raised an event; the operator was mailed for the first; the second's mail was
cooldown-suppressed and that suppression is written down.** The audit is per-retrieval and complete
even where the mail is not.
- **R-203 NEW** — the missing mandatory directory. Highest ID before this session was R-202.
- **R-201** — prepared, halted, with everything it established and exactly what resuming needs.
- **R-200** — plumbing complete (v0.196.0); the form is not built; the install has never run live.
- **R-202** stays open. **The ciphertext deletion is still owed.**
- **The v0.93.0 `identity_blob` retention is still unit-proven only** — nothing here superseded a key.
## 8. Evidence that R persisted nowhere — the search, not a claim
## 13. CI
Searched on the subject host with the pattern read from a file (never in argv), across the agent
journal, the controller's container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`:
Run numbers and task ids in the session summary; **`--no-verify` was not used**.
```
agent journal lines containing R: 0
controller log lines containing R: 0
host files containing R: 0
leftover felhom-idesc-* staging dirs: 0
staged escrow secret file: 0 entries
```
## 14. Observations — noticed, NOT acted on
**With a positive control, because an absent match is not a measurement unless the instrument is shown
to work:** a copy of R was planted under `/tmp`, the same sweep found **1**, the copy was shredded, and
the sweep returned **0**. The instrument detects R when R is there.
R reached the box only as: the credentials-file value on DooPlex → a pipe → the container's stdin →
the request body → the agent's memory. It was never an argument, never a file on either machine, and
appears in no log line at any level.
## 9. Register
- **R-199 → SHIPPED + PROVEN-LIVE.** Links 6, 7 and 8 assembled and walked.
- **R-200 → plumbing shipped; the FORM is not built** and the row stays open for it and for link 9.
- **R-201 → OPEN, and the distinction is deliberate:** the KEY is proven recoverable; a recovered
password has never been **installed**, no repository has been **reopened** under one, and **no file
has ever been restored**. The drill's pass condition is unchanged — a byte-identical sentinel file,
not "the store opened".
- **R-202 → OPEN and untouched.** The orphan card still promises recoverability unconditionally.
- **The orphaned-ciphertext deletion (~1.2 GB) is STILL OWED** — operator ruling 2 of 2026-08-04,
deliberately not ridden along with a code release.
- Capability map, `03-host-agent.md` §8a, `CONTEXT.md` **S-29**, `REUSE.md` and `ROADMAP.md` updated;
`STATUS.md` rewritten for the operator.
## 10. The capability-map row, and what it deliberately does not claim
Added: *"The offsite repository password can be RECOVERED from the sealed escrow with the customer's
recovery code"* — **PROVEN-LIVE (2026-08-04)**, with the evidence above.
**What the row explicitly refuses to claim**, stated in the row itself because the previous over-claim
on this subject was struck out four hours earlier: it covers the **key**, not the **data**. No
recovered password has been installed, no repository reopened, no file restored. And the proof used a
box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is
exactly what the drill covers and it has not run.
## 11. CI
Run numbers, task ids and conclusions for all commits are quoted in the session summary.
**`--no-verify` was NOT used** — every push ran its repo's pre-push gate and passed.
## 12. Teardown
**Nothing was provisioned.** The temporary pattern file used by the R sweep and its positive control
were shredded on the host by the same command that created them (verified: 0 files remain). No
diagnostic artifact persists on either machine. The agent's previous binary is retained as
`/usr/local/bin/felhom-agent.bak-0.124.1`, per the documented deploy.
## 13. Observations — noticed, NOT acted on
1. **The `escrow_blob_served` mail is subject to the 1-hour operator cooldown** — visible above, where
the second retrieval's mail was suppressed. The EVENT rows are per-retrieval so the audit is
complete, but if retrieval ever becomes routine, a burst would produce one mail. Worth revisiting
**when** the customer-facing flow lands, not before.
2. **`ListSupersededEscrow` still has no production caller.** Recovery reads the CURRENT blob; a
customer needing a PREVIOUS generation's key (which is what R-198's retention exists for) has no
path at all. That is the next gap in this area after the drill.
3. **The controller binary is at `/usr/local/bin/felhom-controller`, not `/app/…`** — the first
diagnostic invocation failed on a guessed path. Worth a line in the runbook when one is written.
4. **`age` is at the hardcoded `/usr/bin/age` on both demo hosts** (1.2.1 on demo-felhom) — the recon
listed this as unestablished; it is now established for demo-felhom. `ageBinary` is a package
variable with no config override, so a host that installs `age` elsewhere would fail the unseal at
the last step.
5. **Yesterday's `identity_blob` retention has still not been exercised live** — the operator's fresh
ceremonies sealed the *same* password, so no supersession occurred. It remains unit-proven with a
verified schema; the first real supersession will be its live proof.
1. **`${IMPORT_PATH}` and `${USERDATA_PATH}` resolved to different roots in one deploy** — the sharpest
single expression of R-203, and the place a fix should probably start.
2. **The controller's CSRF form field is `_csrf`, not `csrf_token`** — the login page carries one name
and the protected forms another; an hour went into that. Worth a line in the headless-access memory.
3. **`/api/stacks/<name>/deploy` accepted `HDD_PATH=/mnt/sys_drive`** one call after refusing the NAS
path with a clear Hungarian error. If the system drive is not a valid app namespace, that refusal is
missing.
4. **demo-hp has no registered drive usable as an app namespace** — its only drive is a NAS. Any drill
needing a file-leg app on that box is constrained by this.
5. **The `felhom-r-leak`-style positive control is now used in three separate absence checks** across
two sessions. It is cheap and it has already caught one wrong test; worth making a documented habit.
STATUS.md
+19 -4
View File
@@ -46,6 +46,16 @@ Proven end to end on real hardware.
recovered key back*, reopens the old backup store with it, or **restores a single file**. Today
proves the key survives and returns; it does not prove the backups do. *(R-199 closed; R-200 half;
R-201 open)*
- **A backup reported success while leaving out a folder the customer was told is protected.** Found
this evening on the HP machine, while setting up the wipe-and-restore proof. We deployed an app whose
book folder is declared *mandatory* — the strongest protection class — put a marked file in it, and
ran an off-site backup. **The backup said OK. Three snapshots. The folder was not in any of them.**
The machine knew: one warning line inside the container says the folder was skipped. Nothing else
does — not the card, not the counters, not the hub, not you. This is the same shape as everything
else we have been fixing this month: *a path the customer thinks is protected is not in the copy.*
It is the reason the proof stopped before the wipe — wiping would have destroyed the marked file and
proven nothing. **The two apps that were already backing up off-site are unaffected** — they declare
no such folders. *(R-203)*
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
says the old backups may later be restorable with the matching recovery code. From today that is true
for machines that re-seal from now on and **false for anything already orphaned** — and the machine
@@ -94,10 +104,9 @@ Proven end to end on real hardware.
## What we're working on
- **Now:** the proof exercise — wipe a demo machine and recover it with a saved recovery code. It is
designed, and it is now a much better bet than it was this morning: the first half of the path was
walked live today, so if the drill fails we will know *which* step failed instead of just "recovery
did not work". Waiting on your go-ahead. Both honesty fixes shipped today: a changed backup key now
- **Now:** fixing the folder-left-out-of-the-backup problem above. The wipe-and-restore proof is
**staged and waiting on it** — the machine, the code, the working off-site store, the app and the
marked file are all in place; only the missing folder blocks it. Nothing was wiped. Both honesty fixes shipped today: a changed backup key now
raises an alarm on the day, and the email that stated the opposite of what it measured now describes
what it actually saw.
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
@@ -149,6 +158,12 @@ Proven end to end on real hardware.
## Changed since last update
- **2026-08-04 (late)** — Set up the wipe-and-restore proof on the HP machine and **stopped before the
wipe**: a folder marked as protected was missing from the off-site backup while the backup reported
success. Three things were proved on the way, all firsts: a rebuilt machine's off-site backup now
**refuses and says so** instead of quietly starting over; the "start a new store" repair works and
keeps the old data aside; and the HP machine's pre-3-August history is gone for good — its key was
destroyed four hours before the fix that would have kept it. *(R-203, R-201, R-193)*
- **2026-08-04 (evening)** — **Proved the backup key comes back.** A demo machine fetched its own
sealed package, opened it with the saved recovery code, and produced a key identical to the one it
uses. A wrong code was refused and wrote nothing. The recovery code left no trace anywhere on the
documentation/architecture/00-capability-map.md
+1 -1
View File
File diff suppressed because one or more lines are too long
documentation/audits/DRILL-r201-offsite-recovery-2026-08-04.md
+281
View File
@@ -0,0 +1,281 @@
# DRILL RECORD — R-201, the wipe-and-recover proof: **PREPARED, HALTED BEFORE THE WIPE**
**Date:** 2026-08-04 · **Box:** `demo-hp` (Tier 0, the designated drill host) · **Nothing was wiped.**
**Outcome:** the drill did not reach its verdict. It was halted at step 4 by a defect that makes the
verdict unobtainable — and that defect is worth more than the drill.
> **THE HEADLINE.** A customer-declared **mandatory** data directory was **silently absent from the
> off-site snapshot**, while the backup reported `ok` with three snapshots. The only trace is one
> `[WARN]` line inside the controller container. The hub, the card and the counters all say the backup
> succeeded. → **R-203**
>
> **Nothing irreversible happened.** The wipe never ran. `demo-hp` is left in a BETTER state than it
> started: it now has a working off-site repository (it had an orphaned one), three snapshots, and a
> sentinel file on disk.
---
## 1. The verdict — not reached, and why that is the correct outcome
The drill's pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established that
the sentinel is not in the off-site snapshot at all.** Wiping the box would therefore have:
- destroyed the sentinel, which exists only on that box;
- proven nothing about recovery, because there would be nothing to recover;
- and done so *after* the point of no return.
The runbook's own rule applies: *"If this session finds itself writing code beyond Part 0, stop. That
means a precondition was wrong, and the finding outranks the drill."* A precondition was wrong. It was
one the runbook's P1–P6 table did not contain, because nobody knew to look for it.
**Sentinel sha256 (step 3), recorded and still on the box:**
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt` (181 B).
---
## 2. R-203 — the defect that halted the drill
**Measured, twice, on the live box.**
| what | path | exists? |
|---|---|---|
| the app's live bind, where the customer's files actually land | `/mnt/sys_drive/userdata/media/books` | **YES** (the sentinel is here) |
| the path the off-site capture set treats as the mandatory directory | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
The controller's own log, verbatim:
```
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
/mnt/sys_drive/felhom-data/userdata/media/books
[INFO] [offbox] backed up calibre-web (…/backups/primary/calibre-web, 0 mandatory path(s))
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
```
**The run reported `ok`.** `last_status: ok`, `last_success` stamped, `snapshot_count: 3`. Nothing
customer-visible, nothing hub-visible and nothing in any counter says the mandatory directory was
dropped. This is `CLAUDE.md`'s recurring shape — *a path the customer thinks is protected is not in the
snapshot* — and the code even has the right words for it in a WARN nobody reads.
**The mechanism, from source.** `NamespaceRoot(drivePath, inGuestDrive)`
(`appbackup/paths.go:28-33`) appends `felhom-data` **when the drive IS the system data path**
(`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`,
`backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is documented and computed
as **`<HDD_PATH>/userdata`** (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive`
and an app deployed at `HDD_PATH=/mnt/sys_drive`, those two produce different directories.
**And the same compose file used BOTH roots.** From `docker inspect calibre-web`:
```
bind /mnt/sys_drive/felhom-data/userdata/import/calibre -> /cwa-book-ingest ← felhom-data root
bind /mnt/sys_drive/userdata/media/books -> /calibre-library ← NO felhom-data root
```
`${IMPORT_PATH}` resolved *with* the segment; `${USERDATA_PATH}` resolved *without* it. One deploy, one
template, two roots.
### What is measured and what is not — stated because the scope changes the fix
- **MEASURED:** the two paths disagree; the mandatory directory is absent from the snapshot; the run
reports `ok`; the only signal is a container-log WARN.
- **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a *supported* choice. It was chosen because
demo-hp's only registered drive, `Felhom-Share`, is a NAS and was **correctly refused** as an app
namespace (R-108's `RefuseAsAppNamespace`, working as designed). `/mnt/sys_drive` was **accepted**
(HTTP 202) rather than refused.
**Either branch is a defect, which is why this is filed regardless:**
- if the system drive **is** a supported app namespace → the userdata path resolution is wrong for
every app deployed on it, and their mandatory directories are silently unprotected;
- if it is **not** supported → the deploy accepted a namespace it should have refused, exactly as it
refused the NAS one call earlier, and the refusal that exists is not reaching this case.
**What must NOT be concluded from this drill:** that off-site backups are broken generally. The two
pre-existing apps (`opengist`, `privatebin`) declare **no** mandatory userdata paths — everything they
own is in named volumes — so they are unaffected, and their snapshots are real.
---
## 3. Preconditions, each measured
| # | Precondition | Result |
|---|---|---|
| **P1** | operator holds the recovery code | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file (`~/.config/credentials`). *The code itself appears nowhere in this record.* |
| **P2** | demo-hp on agent v0.125.0 + controller v0.196.0 | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed the published v0.125.0 (sha `f7d8339b53d9…`, verified against the release) and controller v0.196.0. `age` present at `/usr/bin/age`. |
| **P3** | the current `identity_blob` seals the repository under test | **PASS with a caveat that reshaped the drill** — `identity_blob` = 572 B, `restic_pw_sha256` = `8a9e33aa4da6…`. But **no repository existed under that key** (see §4). |
| **P4** | a deliberate, verified whole-guest archive as rollback | **NOT TAKEN** — deliberately. It is only needed for the wipe, and the wipe did not happen. |
| **P5** | demo-felhom untouched and healthy | **PASS** — not touched at any point. |
| **P6** | space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 % used, guest 64 GB free. |
---
## 4. Step-by-step, with every observable
### Step 1 — starting state (hub, read-only)
`demo-hp-bb76ea`: `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, created `2026-08-04T11:11:37Z`, `escrow_state: escrowed`.
Report: `snapshot_count: 0`, `repo_size_bytes: 0`, **no `last_run`/`last_status` at all** — the shape of
a controller that has never run an off-site backup in this lifetime.
### Step 1b — the repository was ORPHANED, and this is the first live proof of a prediction
The 2026-08-04 spike predicted from source — and could not measure — that a rebuilt box's next
off-site run would hit a **third** outcome: neither reattaching the old snapshots nor silently starting
a fresh history, but **refusing**. Measured here, twice over.
**Read-only probe first** (`restic cat config` with the current key, writes nothing):
```
Fatal: wrong password or no key found
```
— the exact string `classifyResticProbe` maps to `"orphaned"`.
**Then the real run, through the customer UI endpoint** (`POST /backup/offbox/run`):
```
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
no-longer-available key; runs will skip until reset
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
[INFO] Event pushed: offbox_repo_orphaned (warning) — A távoli mentési tároló elárvult …
```
`repo_state: "orphaned"`, `orphaned_at: 2026-08-04T12:37:30Z`, `last_status: "error"`, the orphan card
rendered on `/backups/remote`, and the event reached the hub (HTTP 200).
> **The system stopped and said so. It did not quietly start a new history over the old one.**
> That closes R-193's open Q3 — and it is the good half of this month's story.
**Why the repository was orphaned:** the 15 snapshots / 40.9 MB at `/home/felhom-repo` were written
under key `8e03eddf9ff7…` before the 2026-08-03 rebuild. That key survives only in superseded escrow
row **id 3**, which carries **`identity_blob` = NULL** — it was superseded at `2026-08-04 07:15:36`,
**four hours before** hub v0.93.0 fixed the retention. Unrecoverable, permanently, with or without a
recovery code.
### Step 1c — the reset (operator-authorised)
The orphan card's own reset, confirmed by the operator during the session:
```
[WARN] [offbox] resetting orphaned repo (operator-confirmed (claimed)):
move-aside /home/felhom-repo -> /home/felhom-repo.orphaned-20260804, then re-init
[INFO] [offbox] orphaned repo reset complete — old history set aside at
/home/felhom-repo.orphaned-20260804 (move-aside, not deleted); fresh repo initialized
[INFO] Event pushed: offbox_repo_reset (info)
```
**Nothing was deleted.** The reset path had never run in anger before; it works.
### Steps 2–3 — the recovery code and the sentinel
The operator ran the ceremonies for **both** boxes earlier the same day and saved the codes. Measured
hub-side: the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z`
(demo-felhom), and `restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated
them as same-password re-ceremonies: **no superseded row, no `offsite_repo_key_changed`**. That is
v0.93.0's Scenario E, live.
**A file-leg app had to be deployed**, and this is a precondition the runbook did not anticipate:
neither off-site-toggled app on the box (`opengist`, `privatebin`) has a restorable file leg — both
keep everything in **named volumes**, which the off-site tier backs up as tars but the customer restore
flow **never unpacks** (`offbox_reconstitute.go`). A sentinel in either would have been unrecoverable
by design.
`calibre-web` was chosen: it declares `userdata: media/books class: mandatory`, and it is a
single-container app. Deployed through the real API (`POST /api/stacks/calibre-web/deploy`, HTTP 202),
toggled for off-site, and a Tier-1 recovery unit captured
(`Recovery unit captured for calibre-web → …/backups/primary/calibre-web`).
Sentinel written, `sync`ed, hashed: **`643166269103a25c…`**, 181 B.
### Step 4 — the pre-wipe off-site backup: **`ok`, and wrong**
```
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
snapshot_count 3 · repo_size_bytes 30 636 · repo_state null
```
Three snapshots, three apps, 37 s — and **the sentinel is in none of them**, per §2.
### Steps 5–11 — NOT RUN
Step 5 (verify the pre-wipe snapshot contents) would have confirmed the absence a second way; it is
moot given §2. Steps 6–11 (archive, wipe, reinstall, recover, install, restore, compare) were **not
started**. The §7 STOP was never reached, because the drill failed its own precondition first.
---
## 5. Part 0 — shipped, tested, deployed (R-200's plumbing half)
`--recover-offsite-install` (controller **v0.196.0**, commit `1b1366b`): same fetch → unseal → extract
path as `--recover-offsite-check`, same STDIN discipline for R, but it **places** the recovered
password via `InjectOffboxPassword`.
- **Confirmation is a second invocation.** Without `--confirm-install` it prints both hashes and writes
nothing. A single interactive prompt would have had to share stdin with R.
- **Three outcomes, named distinctly:** *installed* (no local password — the rebuilt-box shape),
*unchanged* (identical key already present, nothing written), *refused* (a DIFFERENT key present;
installing would clobber the key the current repository is encrypted under — exit 2, no force offered).
- It re-reads the file after writing rather than trusting the call's return.
**Tests + red-proof.** `go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK.
Removing the confirmation gate makes the dry run write the password and fails
`TestRecoverAndInstall_InstallsOnABareBox` with *"the DRY RUN wrote the password"*. The R-persistence
test carries a **positive control** — a planted copy of the code is found by the sweep, then removed
and not found — because an absence check is worth only what its sensitivity is.
**It was deployed to demo-hp and never exercised against a live recovery**, because the drill halted
before step 9. Its unit proof stands; its live proof does not exist.
---
## 6. What this drill did and did not establish
**Established, live, for the first time:**
1. A rebuilt box's off-site repository is **orphaned and the run refuses** — the spike's predicted
third outcome, measured. It does not silently start a fresh history.
2. The **orphan reset works**: move-aside, never delete, fresh repo initialised, event pushed.
3. **A customer-declared mandatory data directory can be silently absent from the off-site snapshot
while the run reports `ok`** (R-203).
4. demo-hp's pre-rebuild off-site history is **permanently unrecoverable** — its key was destroyed four
hours before the fix that would have kept it.
**NOT established — and unchanged from before this session:**
- **No file has ever been restored from an off-site backup after a wipe.** R-201's question is still
open, and its pass condition is unchanged.
- Part 0's install path has never run against a live recovery.
- The v0.93.0 `identity_blob` retention is **still unit-proven only** — nothing in this session
superseded a key, so nothing exercised it.
---
## 7. State left behind, and teardown
**Deliberately not torn down** — this is evidence, and the box is better off than it was:
| layer | state |
|---|---|
| the guest | `calibre-web` deployed and running, off-site-toggled, sentinel file in place. **Kept** as the drill fixture for the resumed run — it is the only app on either demo box with a restorable file leg. |
| the off-site repository | fresh, working, 3 snapshots, 30 636 B. The old 15-snapshot history is **set aside**, not deleted, at `/home/felhom-repo.orphaned-20260804`. |
| the host | agent v0.125.0, controller v0.196.0, `pvesm` unchanged apart from normal usage. |
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` events recorded for `demo-hp`. No scratch customer records were created — **nothing was reinstalled**. |
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — that act is ruled and still owed,
and §8.3 of the runbook forbids riding it along with a drill. The reset added
`/home/felhom-repo.orphaned-20260804` (≈41 MB) to what is set aside.
**No `R` persisted anywhere** — the recovery code was never used in this session. It was read only to
confirm the key exists in the credentials file; no unseal was performed on demo-hp.
---
## 8. To resume the drill
1. **Fix or scope R-203.** Until the mandatory userdata path lands in the snapshot, no sentinel can
survive a wipe and the drill cannot reach its verdict.
2. Re-run steps 4–5 and confirm the sentinel IS in the snapshot — by listing it, not by a green status.
3. Then P4 (the deliberate archive), the §7 STOP, and steps 6–11 as written.
Everything else is already in place: the code, the versions, the recovery code, the working repository,
the file-leg app and the sentinel.
documentation/backlog/OPEN-ITEMS.md
+3 -2
View File
File diff suppressed because one or more lines are too long
documentation/backlog/ROADMAP.md
+2
View File
@@ -63,6 +63,8 @@
| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
| R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 6–8 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
| R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |