felhom-controller/REPORT.md

# REPORT — R-353 / R-357 / R-358 / R-360: the restore tells the truth

**Controller v0.226.0 · 2026-08-30 · implemented on DooPlex, validated live on `demo-hp`**

---

## 1. Confirmed baselines used — BOTH HAD MOVED

| repo | spec baseline | **actual at start** | version |
|---|---|---|---|
| `felhom-controller` | `f8c9390` / v0.223.0 → v0.224.0 | **`e5eee50` / v0.225.0** | → **v0.226.0** |
| `felhom.eu` | `c2c1fb4` | **`ac6ac03`** | docs only |
| `felhom-agent` | not touched | v0.130.0 | unchanged |

**Both targets were consumed earlier the same day by my own work** — v0.224.0 (R-330) and v0.225.0
(R-331). The drift was re-confirmed against live Gitea before the first edit and the operator
authorised proceeding. **Every symbol in the spec's §5 table was re-verified present at the real
baseline** before editing; all 20 resolved, and almost every line landmark still matched. `MinAgent`
stays **0.129.0**. Both trees were clean and equal to `origin/main` at the start.

## 2. Files created / modified

**`felhom-controller`** (all paths under repo root):

| file | change |
|---|---|
| `controller/internal/backup/restore.go` | `restoreDockerVolumes` → `(int, error)`; caller updated |
| `controller/internal/backup/restore_unit.go` | `UnitRestoreResult`; `RestoreFromRecoveryUnit` → `(UnitRestoreResult, error)` on every return path |
| `controller/internal/web/handlers.go` | `unitRestoreOutcomeMsg` + 3 message constants; handler publishes the outcome |
| `controller/internal/backup/offbox_reconstitute.go` | the R-357 headroom gate |
| `controller/internal/backup/offbox_restore.go` | scratch marker (write/clear/read), `OffboxFullScratchReady` rewritten, shared refusal constants, `SetOffboxLatestSnapshotFn`, `WriteScratchMarkerForTest` |
| `controller/internal/backup/backup.go` | `offboxLatestSnapFn` field |
| `controller/internal/web/offbox_handlers.go` | server-side scratch refusals in 2 handlers; R-360 delete guard; corrected doc comment |
| `controller/internal/backup/r357_reconstitute_headroom_test.go` | **new** |
| `controller/internal/backup/r358_scratch_marker_test.go` | **new** |
| `controller/internal/web/r353_unit_outcome_test.go` | **new** |
| `controller/internal/web/r358_r360_handlers_test.go` | **new** |
| 3 existing `*_test.go` in `internal/backup` | mechanical `_, err :=` for the changed signature |
| `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `controller/README.md`, `REPORT.md` | docs |

**`felhom.eu`**: `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md`,
`documentation/backlog/CLOSED-ITEMS.md`, `documentation/architecture/07-backup-architecture.md`,
`documentation/architecture/00-capability-map.md`,
`documentation/audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (**new**).

## 3. Commits pushed to `main`

| repo | commit | contents |
|---|---|---|
| `felhom-controller` | **`b8af7276`** | all four fixes, all tests, controller docs |
| `felhom.eu` | **`e027b5d9`** | register closures, R-395 fix, architecture, capability map, evidence |
| `felhom.eu` | *(this report's commit)* | housekeeping compression + REPORT |

## 4. Tests and red-proofs

**All pass.** New tests, by group:

| test | result |
|---|---|
| `TestUnitRestoreOutcome_VolumesAndDatabaseNamed` (A1) | pass |
| `TestUnitRestoreOutcome_BackupHeldOnlySettings` (A2) | pass |
| `TestUnitRestoreOutcome_ManifestListedDataThatDidNotReturn` (A3) | pass |
| `TestUnitRestoreOutcome_DatabaseOnly` (A4) | pass |
| `TestR353_HandlerPublishesTheOutcome` (A5, the seam test) | pass |
| `TestR357_DestructiveRestoreRefusesWithoutHeadroom` (B1) | pass |
| `TestR357_UnknownSizeFailsClosed` (B2) | pass |
| `TestR357_UnknownFreeSpaceFailsClosed` (added — the mirror hole) | pass |
| `TestR357_AmpleSpaceIsUnchanged` (B3) | pass |
| `TestR358_FailedRestoreLeavesNoUsableScratch` (C1) | pass |
| `TestR358_UnitOnlyScratchIsNotFullReady` (C2) | pass |
| `TestR358_StaleMarkerIsClearedBeforeTheRun` (C3) | pass |
| `TestR358_UnreadableMarkerFailsClosed` (C4) | pass |
| `TestR358_WrongSchemaFailsClosed` (added) | pass |
| `TestR358_CompletedFullScratchStillReady` (C5) | pass |
| `TestR358_MarkerIsWrittenAt0600AndAtomically` (added) | pass |
| `TestR358_MarkerIsNeverPlaced` (D3) | pass |
| `TestR358_MarkerIsClearedBeforeResticAndWrittenAfter` (added, AST) | pass |
| `TestR358_PlaceHandlerRefusesIncompleteScratch` (D1) | pass |
| `TestR358_ReconstituteHandlerRefusesIncompleteScratch` (D2) | pass |
| `TestR358_UnitOnlyScratchClosesTheFullRestoreCard` (Scenario F, flow level) | pass |
| `TestR360_VerifyCopyDeleteRefusedDuringRestore` (D4) | pass |
| `TestR360_VerifyCopyDeleteStillWorksWhenIdle` (Scenario H) | pass |

**E1 — no existing test was modified for its content.** Every `TestReconstituteOutcome_*` passes
unmodified. The only test-file edits were mechanical call-site updates for the changed
`RestoreFromRecoveryUnit` signature (`err :=` → `_, err :=`) in three files.

### Red-proofs — each mutated, observed failing, reverted, `git diff` clean

| # | mutation | observed failure |
|---|---|---|
| **A5** | `EndRestoreOp(true, stackName+" visszaállítva ("+snapshotID+").")` restored | `THE PRE-FIX SENTENCE REACHED THE CUSTOMER: "opengist visszaállítva (snap-123)."` |
| **B1** | the whole R-357 gate deleted | `THE APP WAS STOPPED (1 call(s)) for a restore with 300 KB free for a 1 MB copy … (err=<nil>)` — and the same on both fail-closed tests |
| **C1** | the old `len(entries) > 0` check restored | `a part-copy was reported READY`, plus the unit-only and unreadable-marker cases |
| **D4** | the delete guard reverted to `IsRunning()` | `THE VERIFICATION COPY WAS DELETED while a restore was writing into it` |
| **D1/D2** | both server-side scratch refusals removed | both handlers redirected with „…elindult" over a part-copy |

> **⚠ THE FIRST B1 RED-PROOF EXPOSED A HOLLOW TEST OF MY OWN, and it is recorded rather than quietly
> fixed.** With the gate removed, the run refused *earlier* — at the placement stat pre-pass — so
> `stops == 0` passed against the pre-fix code and the test proved nothing about the thing it exists
> for. Two corrections: the fixture now populates the scratch the way a completed download leaves it
> (the snapshot's own absolute paths mirrored under the scratch, plus `SetSafetyDumpFn` so no Docker
> is needed), and the assertions are **reordered** so a removed gate reports the outage rather than
> "no error returned". Only after that does the red-proof print `THE APP WAS STOPPED (1 call(s))`.
> The lesson is the doctrine's own: a test that cannot fail on the pre-fix shape is decoration.

**Test count:** 24 new tests added across 4 new files. **Green gate:**
`go build ./... && go vet ./... && go test ./...` → **28 packages, rc 0**, no failures.
**All 12 controller gates OK.**

## 5. Deployed version

```
$ ssh hp "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"
gitea.dooplex.hu/admin/felhom-controller:0.226.0 Up 13 seconds (healthy)
```

**`demo-hp` only.** `demo-felhom` stays on 0.225.0 and the rest of the fleet on 0.223.0 (the floor).

## 6. Live validation — endpoint level, on `demo-hp`

**Method:** the exact endpoints the UI invokes, driven from inside guest 9201 (no browser on
DooPlex; the residual is client-side rendering only). No state was hand-set. Evidence copied off the
box at the end of the phase: `felhom.eu/documentation/audits/evidence-r353-r360-live-2026-08-30/`.

### The verbatim messages

**R-353** — `POST /backup/restore` for `opengist`, then the sentence read off the customer's own
wizard page (`GET /backups/restore/app?name=opengist`):

```
A(z) opengist: 1 adatkötet visszaállítva — az alkalmazás újraindult.
```

and the controller's own lines:

```
[INFO] [backup] Restore-from-unit completed: opengist — 1 volume(s) of 1 listed, 0 database(s) of 0 listed
[INFO] [web]    Restore completed (async): stack=opengist in 9.112668936s (volumes 1/1, dbs 0/0)
```

**This is Scenario A, not Scenario B, and the substitution is stated rather than glossed.** The spec
named `opengist` as the data-less app from 21 August. **It is not one any more** — checked before
relying on it, as the spec instructed: every unit on `demo-hp` today lists at least one volume dump
(`opengist 1/0`, `privatebin 1/0`, `calibre-web 1/0`, the rest 2–3 volumes plus a database). **No app
on the box has the Scenario B shape**, and manufacturing one would mean falsifying a manifest — the
hand-set-state shortcut this project forbids. Scenario B is carried by
`TestUnitRestoreOutcome_BackupHeldOnlySettings` and the A5 seam test. What the live run *does* prove
is the whole path: real counts, correct clause selection (no database clause for 0 databases), and
the sentence reaching the customer's page.

**R-360** — a `mode=unit` scratch restore started, and the verification-copy delete POSTed while it
ran. The refusal, urldecoded from the `Location` header:

```
/backups/restore?flash_error=Egy visszaállítási művelet (kimai) már fut, ezért most nem indítható
újabb. Az állapotát ezen az oldalon követheted; amint befejeződik, újra indíthatsz visszaállítást.
```

```
[WARN] [web] verification-copy delete refused for kimai: a backup/restore op is running
```

**And the consequence, which is the assertion that matters:** a canary file planted in the copy was
still there afterwards — `-rw-r--r-- 1 root root 10 Aug 30 17:34 canary.txt`, contents `defend-me`.

**R-358 / R-396** — after the `mode=unit` restore completed:

```
$ cat …/backups/offsite-restore/kimai/.felhom-restore-complete.json
{"schema":1,"snapshot_id":"84542ec8","full":false,"finished_at":"2026-08-30T17:34:42Z"}
mode=600        (no .tmp left behind)

[INFO] [offbox] kimai: scratch holds a UNIT-ONLY restore (snapshot 84542ec8) — not a full copy,
                so place-to-live stays closed
```

**This is Scenario F proven live**, and it is the case that pre-fix would have unlocked the
destructive restore.

## 7. NOT yet live-validated — awaiting a supervised drill

1. **R-357's disk-full behaviour.** Filling a real filesystem to prove it is a drill step, not a build
   step. Carried by the seam tests, whose central assertion is `StopStack` call count 0.
2. **R-353's Scenario B** (the "backup held only settings" sentence) — no app on `demo-hp` has that
   unit shape any more; see §6.
3. **R-353's Scenario C** (the unit lists dumps, none return) — the R-367 stranded-dump shape was not
   reproduced on live hardware; unit-tested only.
4. **R-358's failed-download branch** was not induced live. It was not needed: the unit-only branch
   exercises the same marker gate through a real restore, without pointing restic at a bad snapshot.

## 8. Teardown

**This run provisioned nothing.** No machine, no guest, no hub record — all three layers:

- **Layer 1 (host):** nothing created on `felhom-pve` or `demo-hp`. No VM, no CT, no storage entry.
- **Layer 2 (guest):** two throwaway shell scripts were pushed into guest 9201 to drive the endpoints
  and **both were removed**; one canary directory was planted under `backups/offsite-restore/kimai`
  on the system drive to prove R-360 and **was removed**. The `mode=unit` scratch restore left a
  normal verification copy on the HDD, which is ordinary product state a customer can delete.
- **Layer 3 (hub):** no customer, appliance or host record created, so none to discard.

`demo-felhom`, `ep0`, DooPlex and Peti's box were not touched.

## 9. Register

**Size: 165 rows before → 167 after filing → 161 after housekeeping.**

- **Closed:** R-353, R-357, R-358, R-360 — each with shipping version and evidence path.
- **Filed and closed in the same session:** **R-396** (Scenario F's answer — see §11) and **R-395**
  (`STATUS.md` contradicting itself; **fixed**, not merely recorded).
- **Re-ranked:** none.
- **Housekeeping:** the six closed rows were compressed into `CLOSED-ITEMS.md` keeping title,
  version, evidence and every sentence that states a rule; the full original is
  `git show e027b5d9 -- documentation/backlog/OPEN-ITEMS.md`. **No open row was touched.**
- **`ROADMAP.md` was not edited:** none of these four ever had a ROADMAP row — they live in the
  register only. Stated rather than silently skipped.

## 10. Documentation coupling (§5.5)

- **`00-capability-map.md`** — one new row, splitting the claim: R-353/R-358/R-360 **PROVEN-LIVE**
  with the evidence citation; **R-357 IMPLEMENTED only**, with the reason.
- **`07-backup-architecture.md`** — §10.2 gained five rows (the four plus R-396). **§8 matrix row 3
  KEEPS its PROVEN status**, with a note recording why: R-353 was a defect in the *message*, not the
  mechanism.
- **`OPEN-ITEMS.md` / `CLOSED-ITEMS.md` / `STATUS.md`** — as §9.
- **`ROADMAP.md`** — no applicable rows.
- **Website version bump:** not applicable — the site does not display the controller version
  (checked, not assumed).

## 11. Observations — each with a register row, or a stated reason it needs none

1. **FILED: R-396.** *Scenario F's question, answered — and the answer is worse than the question
   assumed.* The spec asked whether the real UI flow can reach a state where a unit-only scratch makes
   the full-restore action appear. **It can, by the safest-looking action on the page.**
   „Ellenőrző visszaállítás" (`mode=unit`, the default, advertised as non-destructive) calls
   `RestoreOffboxScratch(full=false)`; `offboxRestoreScratchDir` **ignores `full`**, so both modes
   write the same directory, and `--include` limits *what* restic extracts, never *where*; the wizard
   sets `ScratchReady` from `OffboxFullScratchReady`; `deriveWizardStep` derives **both**
   `PlaceEnabled` and `RestoreEnabled` from that one flag. R-358 as filed assumed the bad state needed
   a *failed* download. It needed only a successful safe one. **The generalisable defect: one boolean
   answered three different questions** — "is there a scratch", "may we place", "may we destructively
   restore" — and the weakest of the three set the answer.

2. **FILED: R-397.** *`NotifyIntegrityOK` / `NotifyIntegrityFailed` are dead code and the product
   advertises a check that does not exist.* Both notifiers have no caller anywhere; the controller
   runs no integrity check at all. **The part that actively misleads:** `config.Monitoring.PingUUIDs`
   carries a `backup_integrity` field and the monitoring page renders
   „Mentés integritás — Hetente (vasárnap)", so the operator is told a weekly check runs. Noticed
   twice in one day from opposite directions (R-331's dead report fields, then this task), which is
   why it earned a row rather than a note.

3. **FILED: R-398.** *`resticStep` is not a seam, so no test can drive any restic-backed path.* Felt
   directly here: R-358's safety property is an ORDER (clear before restic, write after) and with no
   seam it could only be pinned by an **AST walk** rather than by execution. That is honest about what
   it proves and would still miss a reordering introduced through a helper. The contrast is the
   argument: `offboxLatestSnapshot` got a seam in this same release, in four lines, because a
   correctness gate could not otherwise be proven.

4. **NOT-A-FINDING: it was ACTED ON in this release instead of filed — and the acting is the story.**
   `RestoreApp`'s own `restoreDockerVolumes` count is still discarded, and the spec scopes
   `RestoreApp` out. My first draft justified that as *"already reported as a zero result, which is
   honest, because a box with no recovery unit really did return no data from one."*

   > **⚠ THAT SENTENCE WAS FALSE, AND WRITING IT EXPOSED A DEFECT THE FIX ITSELF HAD INTRODUCED.**
   > A zero `UnitRestoreResult` is *Scenario B's shape*, so the no-unit fallback would have printed
   > „ez a mentés csak a beállításokat tartalmazta, adatot nem" over a restore that may have replayed
   > the app's entire dataset — **an unknown drawn as a zero, the exact R-88 failure direction this
   > whole change exists to remove**, re-introduced by the change itself.
   >
   > Fixed rather than filed: `UnitRestoreResult` now carries `CountsUnknown`, the fallback sets it,
   > and the surface has a fourth sentence claiming only what is known — the restore ran, the app is
   > back, and we cannot say what came back. **`RestoreApp`'s signature is untouched**, so §5 holds.
   > Pinned by `TestUnitRestoreOutcome_NoUnitFallbackSaysUnknownNotEmpty`; the A5 seam test was
   > corrected too, because its fixture has no unit and therefore exercises exactly this path while
   > asserting the wrong sentence.
   >
   > **It was the `observations` gate refusing the push that forced the re-read.** The gate exists to
   > stop findings dying in an overwritten REPORT.md; here it caught a live defect instead.

5. **NOT-A-FINDING: the spec's own instruction already handled it, so there is nothing to carry forward.** §13 step 1 named `opengist` as
   the data-less app; it is not one any more. Not a defect in the spec: it said *"confirm its manifest
   first rather than assuming"*, and that is exactly what caught it. Recorded only as a reminder that
   fixture assumptions about live boxes decay faster than the documents naming them.

## 12. Two gates fired; one push was bypassed, and both are declared

**`felhom.eu` — `golden-currency` CONVICTED.** Three controller releases shipped today (0.224.0,
0.225.0, 0.226.0) and the vouched golden still carries **0.223.0**, so a machine installed right now
receives none of them. **The gate is right.** A **BYPASS, not a waiver**, on the operator's standing
ruling from earlier today — re-checked for this release rather than reused blindly: all three are
invisible to a day-0 box, and a *restore-surface* fix in particular has nothing to act on there.
**The ground expires the moment a release changes first-boot behaviour.** Tracked on **R-242**; **one
bake carrying 0.226.0 covers all three**, then raise the floor to 0.226.0.

**`felhom-controller` — nothing was bypassed.** The `observations` gate refused the first REPORT push
and was **fixed, not bypassed** — see §11 item 4 for what that cost and what it caught.

**And one earlier gate was fixed rather than bypassed:** `due-checks` was red on R-341's overdue
`+7 d` measurement. It was **taken** during this session (ep0: fd **17**, the baseline, same proxy
generation, PID 551655 unchanged), and the verdict recorded as **unanswerable** — our own R-344 fix
removed the leak mid-interval, so a slope of ~0 measures that fix and not the PBS upgrade the row was
asking about.