felhom.eu/REPORT-r204-item4.md

# REPORT — R-204 item 4 / R-193 credential half (hub v0.96.0), 2026-08-05

**A rebuilt box asks for its credential back, and the hub answers.** The last of the four manual
interventions the 2026-08-04 drill needed. Controller half: `felhom-controller` v0.199.0.

> **Written as `REPORT-r204-item4.md`, not `REPORT.md`.** A PARALLEL SESSION is active in this shared
> clone — it committed `c917251` (the R-205…R-211 disk containment) between my baseline read and my
> first commit. Per `CLAUDE.md`'s parallel-session rule the second session never touches the shared
> `REPORT.md`. Explicit per-file staging was used throughout; verified after the fact that none of my
> three commits carries a foreign file.

## 1. Baselines, re-read on arrival — with a drift

| Repo | Expected | Found |
|---|---|---|
| `felhom.eu` | `0dbd954fec90` / hub v0.95.0 | **DRIFTED to `ee9d9bf`** — one docs-only commit ahead (R-205…R-211 spike output). Hub code, `manifests/hub.yaml` and the deployed image were all still v0.95.0, so the drift did not affect the work. A second foreign docs commit (`c917251`) landed mid-session. |
| `felhom-controller` | `68f195676b91` / v0.198.0 | exact match, tree clean |
| `felhom-agent` | v0.125.0 | `3f5f61b`, clean (rider only, **no version bump**) |
| `app-catalog-felhom.eu` | n/a | `122bbee`, clean (rider only, **no version bump**) |

**The task's "highest register ID in use: R-204" was STALE.** Grepped before minting, as instructed:
the highest is **R-211**. This session mints **R-212** (§8 below).

## 2. Part 2.0 FIRST — does the stored value survive a consume? **YES.**

Established from the schema and the code, deliberately **not** from the PBS analogy:

- `one_time_secrets` declares `value TEXT NOT NULL` (store.go, the CREATE TABLE).
- `ConsumeOneTimeSecret` runs `UPDATE one_time_secrets SET consumed_at = datetime('now')` — **it
  touches nothing else.** The value column is never cleared or overwritten.

**So restage-before-mint is possible**, and `RestageOneTimeSecret` mirrors `RestageHostPBSSecret`:
clear the consumed flag, never touch the value, never bump a generation, return false when no row
exists so the caller escalates.

This is asserted rather than assumed by `TestRestageOneTimeSecret_ReArmsTheSameValue`, which checks
the **same** value comes back — so a future hardening that cleared the column fails loudly instead of
silently turning every rebuild into an external mint. The two secrets are different objects with
different lifecycles, and assuming a shared shape is how two earlier sessions confused the credentials.

## 3. The declared state — name, shape, and how a pre-upgrade hub treats it

**Name:** `needs_credential`, on the report's existing `offsite` object (not a new top-level field).
Constants: `backup.OffsiteStateNeedsCredential` / `offsiteheal.StateNeedsCredential`, pinned to each
other by `TestDeclaredStateStringMatchesTheReconciler`.

**Shape, as received by the hub live** (report id 16743):

```json
{"enabled": false, "escrow_state": "", "state": "needs_credential",
 "snapshot_count": 0, "repo_size_bytes": 0, "quota_gb": 0}
```

**A pre-upgrade hub treats it as INERT, established from the readers' code rather than assumed:**

| Reader | Behaviour on the declaration | Why |
|---|---|---|
| `OffsiteChecker.isStale` | no staleness alarm | returns early on `!off.Enabled` |
| `OffsiteChecker.fillBand` | `bandOK` | returns OK on a zero quota/size |
| `encoding/json` | ignores `state` | unknown field, not an error |
| **`reportHasOffsite`** | **would have MISREAD it** | see below — the one reader that needed changing |

**A configured box's report JSON is byte-identical to v0.198.0's** — `state` is `omitempty` and is
never set on a configured box (`TestOffsiteDeclare_ConfiguredBoxJSONIsUnchanged`).

## 4. The debounce: TWO distinct reports, derived not chosen

The controller reports every ~15 minutes. Two distinct declarations mean the state survived a full
report cycle, and a restart, a slow first report or a transient config read all resolve well inside
one. **One** would act on a blip; **three** would leave a genuinely stranded customer waiting ~45
minutes for the one thing they cannot obtain any other way. The reconciler's own sweep is 5 minutes —
deliberately faster than the report cadence so it adds no latency of its own — and the debounce counts
**fresh evidence, not ticks**, so a fast sweep cannot shorten it.

## 5. §8.4 — is there a deliberately-unhealed state here? **Yes, and it is excluded upstream.**

`pbsdrheal` refuses to heal `verify_failed` because re-staging would not help it and it must stay
loud. The off-site analogue is the **regressed** shape: a box that HAD a working tier and lost its
target while still holding its repository password. Re-arming a credential would not help it either.

**It cannot reach this reconciler at all** — the controller's declaration predicate requires the
repository password to be **absent**, so a box that still holds one never declares. The unhealable
case is excluded *by construction*, upstream, rather than filtered out in a switch. Stated explicitly
because "there is nothing like that here" is usually wrong, and this was checked rather than assumed.

## 6. Files modified

| File | Change |
|---|---|
| `hub/internal/store/store.go` | **new** `RestageOneTimeSecret`, **new** `LatestReportOffsiteDeclaration`; `reportHasOffsite` tightened to require `enabled:true` (+ its comment corrected) |
| `hub/internal/offsiteheal/reconciler.go` | **new package** — the reconciler |
| `hub/internal/offsiteheal/{reconciler,wiring}_test.go` | **new** — Scenarios A–F + the wiring AST test |
| `hub/internal/store/offsite_restage_test.go` | **new** — the value-survives proof + the `reportHasOffsite` equivalence table |
| `hub/internal/monitor/offsite_delivery.go` | `shapeDeclared` outranks both inferred shapes; `maybeHeal` stands down with a record |
| `hub/internal/monitor/offsite_declared_test.go` | **new** — the demo-hp shape + the cross-package string pin |
| `hub/cmd/hub/main.go` | wires + runs the reconciler (`OFFSITEHEAL_ONLY_CUSTOMER` for a supervised rollout) |
| `.githooks/pre-push` | the workspace-root assertion (rider) |
| `manifests/hub.yaml` | image `0.95.0` → `0.96.0` |

**Commits on `main`:** `f62a115` (hub half) · `fe1e816` (rider) · `b2462b8` (CHANGELOG) ·
`4114c5f` (manifest). **Deployed:** `felhom-hub:0.96.0`, ArgoCD **Synced/Healthy**, `deploy/hub`
rolled out, startup log carries `offsite credential self-heal reconciler started (interval 5m,
debounce 2 reports)`.

## 7. Tests and red-proofs

Green gate: `go build ./... && go vet ./... && go test ./...` in `hub/` — **rc=0**.
`python3 scripts/repo_gates.py --fast` — **all five gates OK**.

| Scenario | Test | Result | Red-proof — what was mutated | Outcome |
|---|---|---|---|---|
| A+D | `TestScenarioAD_DeclaringBoxIsRestagedNotMinted` | PASS | inverted `heal()` so Reissue runs before Restage | **FAILED** — *"a sustained declaration was not served: restages=0"* |
| B | `TestScenarioB_SilentBoxIsNeverActedOn` | PASS | — | — |
| C | `TestScenarioC_HealthyBoxIsAPureNoOp` | PASS | — | — |
| E | `TestScenarioE_NothingToRestageEscalatesOnce` | PASS | — | — |
| F | `TestScenarioF_BlipIsAbsorbedByTheDebounce` | PASS | `debounceReportsDefault` 2 → 1 | **FAILED** — *"a one-report blip triggered a credential action"* (and Scenario A+D also failed, *"acted on a SINGLE declaration"*) |
| H | `TestMainWiresTheOffsiteHealReconciler` | PASS | (AST; comments dropped) | — |
| — | `TestRestageOneTimeSecret_ReArmsTheSameValue` | PASS | — | — |
| — | `TestReportHasOffsite_EnabledOnly` | PASS | — | — |
| — | `TestShapeOf_DeclarationOutranksBothInferredShapes` | PASS | — | — |
| — | `TestDisabledDescriptorIsNeverHealed`, `TestBlockedCustomerIsNeverHealed`, `TestRestageErrorDoesNotEscalate` | PASS | — | — |

**A test that first passed for the wrong reason, caught and fixed.** `TestBlockedCustomerIsNeverHealed`
initially seeded `Status: "blocked"` through `SaveCustomerConfig`, whose INSERT does not carry the
column — so the customer was never actually blocked and the assertion would have been vacuous. It now
goes through `SetCustomerConfigStatus` **and asserts `IsCustomerBlocked` before proceeding**.

## 8. Part 4 — STOPPED, corrected, then COMPLETED with the operator's confirmation

§8.7: *"If the paths do not match R-193's record exactly, STOP. A near-match on a protected endpoint
is not a match."* **They do not match.**

Measured read-only over SFTP, using each box's own credential, from inside its guest:

| Customer | Path | Size | What it is |
|---|---|---|---|
| demo-felhom (`u629488-sub1`) | `/home/felhom-repo` | **1.2 G** | **LIVE** — the configured `repo_path`. Unopenable by the box (R-193), but NOT a set-aside store |
| demo-felhom | `/home/felhom-repo.orphaned-20260717` | **1.4 G** | set aside |
| demo-felhom | `/home/felhom-repo.orphaned-20260718` | **3.0 M** | set aside |
| demo-hp (`u629488-sub3`) | `/home/felhom-repo` | **582 K** | **LIVE** |
| demo-hp | `/home/felhom-repo.orphaned-20260804` | **43 M** | set aside |

The ruling says *"~1.2 GB across the two demo boxes, in set-aside stores"*. Reality: **three** set-aside
stores totalling **~1.45 GB** — and **the figure that matches ~1.2 GB is demo-felhom's LIVE
`felhom-repo`**. Had the size been used to identify the target, the live repository would have been
deleted. Filed as **R-212**, and the operator was asked with the corrected list.

**The operator confirmed *delete all three*, and all three were deleted.**

| Account | Deleted | Freed |
|---|---|---|
| `u629488-sub1` | `felhom-repo.orphaned-20260718` | 3.0 M |
| `u629488-sub1` | `felhom-repo.orphaned-20260717` | 1.4 G |
| `u629488-sub3` | `felhom-repo.orphaned-20260804` | 43 M |

**AFTER, on each account, a full listing:** `u629488-sub1` holds `.ssh` + `felhom-repo` (**1.2 G**,
live); `u629488-sub3` holds `.ssh` + `felhom-repo` (**582 K**, live). **Nothing outside the three named
paths was touched**, and no prune job, datastore or live repository was involved.

**Proof nothing live was caught:** a REAL off-site run triggered on demo-hp immediately afterwards
(`POST /backup/offbox/run`, authenticated + CSRF) returned `status: "ok"`, `orphaned: false`,
`last_error: ""`, `last_run: 2026-08-05T09:14:02Z`, `last_duration: 1m24s`, 6 snapshots.

**METHOD NOTE, worth carrying forward.** The Hetzner storage box runs a **restricted shell**: no shell
operators, no `test`, no GNU long flags. The first attempt used `test -d X && rm -rf -- X` and got
*"Command not found. Use 'help' to get a list of available commands."* — **it failed CLOSED, verified
by a byte-identical before/after listing.** `rm -r <path>` issued as ONE simple command is the working
form, and the smallest store was deleted first to confirm the syntax before the 1.4 GB one.

## 9. Live validation

| # | What | Observable |
|---|---|---|
| 1 | **A healthy box: no action, no events** | demo-hp reported healthy throughout (`enabled:true`, no `state` key — report id 16742). Zero `offsite_selfheal_*` rows in the hub DB. **Honest limit:** the reconciler is silent by design on a healthy sweep, so there is no per-tick positive observable; what I have is the startup line proving `Run` was entered, the DB showing no events, and Scenario C. |
| 2 | **The declared state, produced live without wiping a box** | demo-felhom 9201 arranged **reversibly** into the stranded shape (settings + `offbox/` backed up first; the `offbox` key removed and `repo_password` moved aside). Report **id=16743** reached the hub carrying `{enabled:false, state:"needs_credential", quota_gb:0, repo_size_bytes:0}`. Restored the same minute; report **id=16744** is healthy again. **The single declaration was absorbed by the debounce — no self-heal event fired** — which is Scenario F demonstrated on live infrastructure rather than in a fake. |
| 2b | **The ACK field is no longer discarded** | Both demo boxes' `settings.json` now carry `hub_escrow_identity_present = true` — the recorder working on a HEALTHY box, which is the case that used to return early. |
| 3 | **A push from outside the workspace is refused** | A scratch clone at `/tmp/.../outside-clone`: `pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace`, naming `/mnt/5_hdd/felhom.eu`, **before the gates run**. **Red-proof:** with the assertion removed the same push **succeeded** (`rc=0`, new branch on a throwaway bare remote). In-workspace pushes ran normally all session. |
| 4 | **Part 4** | **Completed after the STOP and a corrected list.** Full listings before and after on both accounts; both LIVE repos intact; a real off-site run on demo-hp succeeded immediately afterwards (`status: ok`, `last_run 09:14:02Z`). |

**Not fired live: the hub actually re-staging a credential.** Doing so would have re-applied
demo-felhom's off-site target mid-session and changed the very state Part 4's listing describes. It is
proven by Scenarios A/D/E with the store primitive proven separately against a real SQLite database.

**Teardown:** the scratch clone and throwaway remote are removed; demo-felhom's `settings.json` and
`offbox/` restored from the backup taken first (verified: `enabled=True`, `escrow_state=escrowed`,
`repo_password present=True`); the backup copy remains at `/root/r204-backup` on the guest for
traceability. Nothing else was provisioned.

## 10. Registers

- **R-204 — ALL FOUR ITEMS CLOSED.** Both 2026-08-05 rulings recorded on the row: the declared-state
  trigger with its four-meanings-of-absence reasoning, and that the recovery preview's
  dashboard-password exposure is **metadata, not content, and accepted**.
- **R-193 — credential half CLOSED.** Still open under this ID: the customer-facing recovery **screen**,
  and the ciphertext deletion (now R-212).
- **R-192 — CLOSED, guard half by REPLACEMENT.** The counting inference is outranked by the declaration.
- **R-202 — untouched, still open.**
- **R-212 — NEW** (R-211 was the highest; grepped): the halted deletion, with the full measured listing.

## 11. CI

felhom-agent **0404f60**, app-catalog **ee2c810**, felhom-controller **992803c**, felhom.eu
**4114c5f** — run IDs and conclusions confirmed in the controller's `REPORT.md` §11 and re-checked at
session end. **`--no-verify` was NOT used**; every push ran the pre-push gate, including the new
workspace-root assertion.

## 12. Observations — noticed, NOT acted on

- **A second, accidental barrier exists outside the workspace and should not be relied on:**
  `felhom-agent`'s `reuse-refs` gate FAILS in a clone outside the workspace because the shared
  `reuse_refs_check.py` lives in the `felhom.eu` sibling. That is why the first Scenario-G red-proof
  had to be redone with `app-catalog-felhom.eu`, whose fast gate is self-contained. It is incidental,
  repo-specific and not a substitute for the assertion.
- **`offsite_credential_restaged` (the R-71c event) has still never fired for any customer.** The new
  reconciler emits its own `offsite_selfheal_*` events, so the old one may now be permanently dead —
  worth a deliberate look rather than leaving two event families for one concern.
- **demo-felhom's live `felhom-repo` is 1.2 GB of ciphertext its own box cannot open** (0 snapshots,
  `last_status: error`). It is not a set-aside store, so it is out of R-212's scope as filed — but it is
  the largest single block of unrecoverable data in the fleet and nothing currently plans its disposal.