353 lines
19 KiB
Markdown
353 lines
19 KiB
Markdown
# REPORT — R-87: the box proves its own off-site copy still holds something (2026-08-31)
|
||
|
||
Controller **v0.231.0** · hub **v0.110.0** · both deployed and verified live on `demo-hp`.
|
||
|
||
---
|
||
|
||
## 1. Baselines, re-checked at the start
|
||
|
||
| repo | `main` @ | matched the task's stated baseline? |
|
||
|---|---|---|
|
||
| `felhom-controller` | `2d802d75e88616d86cbade8a0e16965c2b85771c` (v0.230.0) | **yes** |
|
||
| `felhom.eu` | `177c75781e11e24fddbaf73c2159f1efd82479e4` | **yes** |
|
||
| `felhom-agent` | `058b9450648a359856a4102bea4650e33d8884cd` (v0.130.0) | **yes**, untouched |
|
||
|
||
All three trees clean, `HEAD == origin/main`. `MinAgent` stays **0.129.0**.
|
||
|
||
---
|
||
|
||
## 2. §5's two answers
|
||
|
||
### 5.1 The acceptance rule
|
||
|
||
**Two parts, and part 1 alone is the trap.**
|
||
|
||
1. everything the manifest declares is present in the restored unit, **and**
|
||
2. **the manifest declares what the app is supposed to have.**
|
||
|
||
The spike's own summary — "check it against its own packing list" — is part 1, and taken literally it
|
||
**passes a hollow unit**, because a hollow unit declares nothing. That is exactly the shape the job
|
||
exists to catch. Part 2 is the whole value.
|
||
|
||
`JudgeRestoredUnit` (`r403_hollow.go`) returns **three** outcomes: `pass`, `fail`, `cannot_judge`.
|
||
|
||
### 5.2 Where the expectation comes from — and the volume half WAS built
|
||
|
||
**From inside the unit, never from the live box.** The snapshot may predate the app's current shape,
|
||
and `GetDockerVolumes` (`backup.go`) enumerates from live Docker, which answers a different question.
|
||
|
||
| half | source | built? |
|
||
|---|---|---|
|
||
| database | `DBServiceNames(composePath)` on the unit's own captured compose — the same discriminator `RestoreFromRecoveryUnit` uses, so this cannot disagree with the restore path about what an app is | **yes** |
|
||
| volumes | `ParseComposeNamedVolumes(composePath)` on the same file | **yes, as an EXISTENCE check** |
|
||
|
||
**The volume half was BUILT, not deferred, and it is deliberately not a name match.** §5.2 asked me to
|
||
establish whether the unit's compose can answer it before building on it. It can — measured, not
|
||
assumed, on all eight real units on `demo-hp`:
|
||
|
||
```
|
||
bookstack 2 tars / 2 compose volumes opengist 1 / 1
|
||
docmost 3 / 3 privatebin 1 / 1
|
||
kimai 2 / 2 calibre-web 1 / 1
|
||
romm 3 / 3 paperless-ngx 3 / 3
|
||
```
|
||
|
||
and `<stack>_<volume>.tar` held in every case. **But "held on eight" is not "derivable":** volume tars
|
||
are `<project>_<volume>.tar` and `ResolveDockerVolumeNames` derives the project from
|
||
`filepath.Base(filepath.Dir(composePath))`, which **inside a unit is the literal string `compose`**,
|
||
not the stack. So the existence question (*does the compose declare named volumes → the manifest must
|
||
declare at least one tar*) needs zero inference and was built; name-level matching needs the
|
||
project-prefix inference and was not. R-355's rule: a claim about the app must never be inferred from
|
||
a counter. **Half a rule that is true beats a whole rule that is invented.**
|
||
|
||
---
|
||
|
||
## 3. Files created / modified
|
||
|
||
**Created:** `internal/backup/offbox_proof.go`, `internal/backup/r87_judgement_test.go`,
|
||
`internal/backup/r87_proof_job_test.go`, `internal/backup/r87_wiring_test.go`,
|
||
`internal/notify/r87_proof_event_test.go`.
|
||
|
||
**Modified (controller):** `internal/backup/r403_hollow.go` (the judgement, beside the R-403
|
||
predicate it reuses), `internal/backup/offbox_restore.go` (`offboxScratchDirIn` +
|
||
`unitOnlyHeadroom`), `internal/backup/offbox_verify_copies.go` (`offsiteProofRootFor`),
|
||
`internal/backup/offbox.go` (report fields), `internal/settings/settings.go`,
|
||
`internal/notify/notifier.go`, `internal/web/handler_debug.go`,
|
||
`internal/web/templates/debug.html`, `cmd/controller/main.go`, plus `CHANGELOG.md`, `CONTEXT.md`,
|
||
`REUSE.md`, `controller/README.md`.
|
||
|
||
**Modified (felhom.eu):** `hub/internal/api/handler.go`, `hub/internal/notify/dispatcher.go`,
|
||
`hub/CHANGELOG.md`, `manifests/hub.yaml`, `scripts/wire_contract_gate.py`, the capability map,
|
||
`07-backup-architecture.md`, `ROADMAP.md`, both registers.
|
||
|
||
---
|
||
|
||
## 4. Commits pushed to `main`
|
||
|
||
| repo | commit | what |
|
||
|---|---|---|
|
||
| `felhom.eu` | `1aeaa30` | hub v0.110.0 — allowlist + operator-only + the wire-contract allowlist |
|
||
| `felhom-controller` | `e43b5ec` | v0.231.0 — the judgement, the job, the alarm, 33 tests |
|
||
| `felhom-controller` | `303129e` | the by-hand trigger |
|
||
| `felhom.eu` | *(this report's commit)* | evidence, capability map, architecture, registers |
|
||
|
||
**The hub shipped FIRST and that ordering is load-bearing:** an event type the hub does not allowlist
|
||
is answered **400 and vanishes**. Deploying the controller first would have made every alarm silent
|
||
until the hub caught up.
|
||
|
||
---
|
||
|
||
## 5. Tests and the red-proofs
|
||
|
||
**33 new tests**, all green. Full controller suite **1689 tests / 28 packages / rc=0**; hub suite **18
|
||
packages / rc=0**. All 13 controller gates and all 13 `felhom.eu` gates OK except the declared
|
||
golden-currency debt (§8).
|
||
|
||
**Five red-proofs, run and recorded — each with the wrong value visible:**
|
||
|
||
| # | what was broken | result |
|
||
|---|---|---|
|
||
| **A2** | the rule replaced by "every declared file is present" (§5.1's trap) | `TestR87_HollowUnitForAnAppWithADatabaseFAILS` FAILED: **`got verdict="pass"`** — and two siblings failed with it |
|
||
| **A3** | the expectation source dropped; alarm on any empty unit | `TestR87_AppWithNoDatabaseAndNoVolumesPasses` FAILED: **`got verdict="fail" reason="database_expected_none_captured"`** |
|
||
| **B2** | the deferred scratch delete removed | `TestR87_ScratchIsDeletedOnEveryPath` FAILED on **all four** rows: *"…/backups/offsite-proof/kimai still exists"* |
|
||
| **B3** | the restore routed the customer path's way (`unlockStale` + `resticStep`, no `--no-lock`) | `TestR87_NeverWritesToTheRepository` FAILED: **`the proof issued a WRITE verb "unlock"`**, with the argv printed |
|
||
| **B5** | `ProvedSnapshots` recording `time.Now()` | `TestR87_ProvedSnapshotIsRecordedNotATimestamp` FAILED (**`got "2026-08-31T18:36:47Z"`**) **and so did the rotation** — *"night 2 re-picked bookstack"*, which is the real consequence |
|
||
|
||
**Three further red-proofs on the wiring**, because an AST test that matches nothing passes silently:
|
||
dropping the `sched.Daily` registration, emptying its closure, and un-guarding the alarm each failed
|
||
`TestR87_JobIsRegisteredInMain` / `TestR87_RunnerAlarmsOnlyOnFail` by name.
|
||
|
||
Every red-proof was reverted and the file confirmed **byte-identical** afterwards.
|
||
|
||
**Test count: 1656 → 1689 (+33).** *(An earlier `git stash` comparison produced a nonsense "545
|
||
before" — stashing the tracked edits left the new untracked files behind, so the tree did not build
|
||
and packages silently failed to list. Counted directly instead; the instrument was the problem.)*
|
||
|
||
---
|
||
|
||
## 6. Deployed version
|
||
|
||
```
|
||
demo-hp guest 9201: gitea.dooplex.hu/admin/felhom-controller:0.231.0 Up (healthy)
|
||
hub (k3s): gitea.dooplex.hu/admin/felhom-hub:0.110.0 Synced / Healthy
|
||
```
|
||
|
||
---
|
||
|
||
## 7. The five live validations — endpoint level, on `demo-hp`
|
||
|
||
Method: `POST /api/debug/backup/offsite-proof`, the exact endpoint the debug button invokes. No
|
||
browser on DooPlex; only client-side rendering is unexercised.
|
||
|
||
### 7.1 The good case — **PASS**
|
||
```
|
||
{"stack":"bookstack","snapshot":"91154be7","verdict":"pass","duration_ms":2913}
|
||
[INFO] proof: bookstack PASSED on snapshot 91154be7 in 2.913s
|
||
```
|
||
**2.913 s against the spike's measured 2.3–4.0 s band.** The scratch was **gone** afterwards, the
|
||
verdict persisted (`proved_snapshots {'bookstack': '91154be7'}`, `last_proof_result 'pass'`), and the
|
||
customer's own verification copies were untouched — including `bookstack`'s, which sat in
|
||
`backups/offsite-restore/` throughout.
|
||
|
||
### 7.2 The case that matters — **the hollow backup was CAUGHT**
|
||
```
|
||
[ERROR] proof: opengist on snapshot f32e1078 is READABLE AND EMPTY
|
||
(volumes_expected_none_captured: opengist_data)
|
||
— the store is not damaged; the backup does not contain this app's data
|
||
|
||
Event pushed: offsite_proof_empty (error) — A(z) opengist legutóbbi távoli mentése olvasható,
|
||
de nem tartalmazza az alkalmazás adatait. A tároló nem sérült — a mentés készült el üresen.
|
||
A mentést újra el kell készíteni; addig ebből a mentésből nem lehet visszaállítani.
|
||
PushEvent: offsite_proof_empty pushed OK (HTTP 200)
|
||
```
|
||
**Exactly one** event (`grep -c "Event pushed: offsite_proof_empty"` → **1**), severity **`error`**,
|
||
and the hub answered **HTTP 200** — which is itself the proof the allowlist entry landed, because an
|
||
unallowlisted type is 400'd. Verdict persisted with its reason. **Scratch deleted on the failure path
|
||
too.**
|
||
|
||
**HOW THE SHAPE WAS PRODUCED — the natural route was tried FIRST and it failed.** I stopped
|
||
`opengist`, on the reasoning that a stopped app is one of R-403's own named causes (a failed dump
|
||
leg), and removed its volume tar. **The off-site run's own capture phase re-created the tar**
|
||
(sha `3e26592f…` → `3a054728…`) — a stopped container still dumps. So:
|
||
|
||
> **DECLARED CONSTRUCTION.** The hollow unit was built by hand: the real compose copied verbatim (so
|
||
> it still declares `opengist_data`, the expectation source) with a manifest declaring
|
||
> `db_dumps: []` and `volume_dumps: []` — the manifest a capture writes for a unit that lost its
|
||
> dumps. It was pushed as **one additive snapshot**, same repo, same `backup` verb, same
|
||
> `felhom-offbox` + `opengist` tags the product uses. **No forget, no prune, nothing deleted.** The
|
||
> healthy history stayed. **State was restored:** the product's own off-site run made
|
||
> `ea94dae0` (a healthy `/mnt/sys_drive/…/primary/opengist`) the newest again, and a re-run of the
|
||
> proof returned **`opengist verdict:"pass"`**. The constructed tree was removed.
|
||
|
||
### 7.3 The healthy control — **PASS, five times**
|
||
`bookstack`, `calibre-web`, `docmost`, `kimai`, `opengist` all passed, 2.2–4.0 s each.
|
||
**`calibre-web`, `opengist` and `privatebin` have no database at all**, so the no-database branch of
|
||
Scenario C is proven live, not only in a fixture.
|
||
|
||
> **What is NOT live-proven, stated rather than glossed:** no app on `demo-hp` has **neither** a
|
||
> database **nor** a named volume, so the exact neither/nor instance of Scenario C has no live
|
||
> subject. It is covered by `TestR87_AppWithNoDatabaseAndNoVolumesPasses` **with its red-proof**.
|
||
|
||
### 7.4 The read-only proof — **no lock, no write verb**
|
||
The lock sampler was **positively controlled before it was believed**: across a real
|
||
`restic check` it went `locks=0 → locks=1` for nine consecutive samples `→ locks=0`. Against the
|
||
proof, run in isolation:
|
||
```
|
||
19:14:36 locks=0
|
||
19:14:40 locks=0
|
||
19:14:44 locks=0 | restic … restore b5aa8f9b --target …/offsite-proof/opengist --include …
|
||
19:14:48 locks=0
|
||
```
|
||
**Zero locks, with the restore caught in flight.** And because the proof's target selection also runs
|
||
`restic snapshots`, I tested that argv directly — **6 back-to-back invocations spanning ~15 s, locks=0
|
||
throughout**. `restic snapshots` does not lock in 0.14.0 either.
|
||
|
||
> **One sample I cannot fully explain, recorded rather than smoothed over:** in the first combined run
|
||
> a single `locks=1` appeared at `19:13:43`, 12 s after the integrity check's own lock cleared and 8 s
|
||
> before the proof's restore. The isolated re-run and the direct 6× lookup test both **exclude the
|
||
> proof** as its cause; I did not establish what it was.
|
||
|
||
### 7.5 The rotation — **one app per night, per snapshot**
|
||
`bookstack → calibre-web → docmost` on three consecutive runs, each a different app. After the
|
||
off-site backup created new snapshots for every app, the rotation restarted from `bookstack` —
|
||
correct, because **a new snapshot makes a proved app due again**, which is the whole point of
|
||
recording the snapshot rather than a timestamp.
|
||
|
||
### 7.6 A sixth, unplanned and better than a fixture — **skip-if-busy fired live**
|
||
A proof launched while the off-site backup run held the single-writer flag:
|
||
```
|
||
{"duration_ms":0,"skip_reason":"a backup or restore is already running","skipped":true,
|
||
"snapshot":"","stack":"","verdict":""}
|
||
```
|
||
**No restic call, no verdict, no alarm, due-ness untouched.** Scenario F1 on real hardware.
|
||
|
||
---
|
||
|
||
## 8. The schedule slot, and the live times it was chosen from
|
||
|
||
**05:30**, read off the running box rather than a document:
|
||
|
||
| job | slot | measured |
|
||
|---|---|---|
|
||
| db-dump | 02:30 | — |
|
||
| tier2-backup | 03:30 | — |
|
||
| **offbox-backup** | **04:15** | 2m52s |
|
||
| offsite-abandon-sweep | 05:10 | — |
|
||
| **offsite-proof** | **05:30 ← new** | one app 2.2–4.0 s |
|
||
| offsite-integrity | 06:00 | 40.3 s |
|
||
|
||
Confirmed registered on the box:
|
||
`Daily job offsite-proof scheduled for 2026-09-01 05:30:00 CEST (waiting 8h19m45s)`, `totalJobs=14`.
|
||
|
||
---
|
||
|
||
## 9. NOT yet live-validated — explicit
|
||
|
||
1. **The unattended nightly firing.** The job is REGISTERED; that is not the same claim. First real
|
||
firing 2026-09-01 05:30 CEST.
|
||
2. **The fleet.** `demo-felhom` is on 0.230.0 and does not have this job. Only `demo-hp` was deployed.
|
||
3. **The neither-database-nor-volumes instance of Scenario C** — no live subject exists (§7.3).
|
||
4. **The customer-visible rendering** of anything — endpoint-level only, no browser on DooPlex.
|
||
5. **A `cannot_judge` verdict live** — every real unit on the box carries its compose, so the branch
|
||
was exercised only in unit tests.
|
||
|
||
---
|
||
|
||
## 10. Capability map, and what did NOT move
|
||
|
||
**Added:** a `PROVEN-LIVE` row for *"the box proves its own off-site copy still holds something"*,
|
||
citing `documentation/tests/r87-offsite-proof-2026-08-31/`, with the scheduled firing marked
|
||
**IMPLEMENTED only**.
|
||
|
||
**`07-backup-architecture.md` §8 matrix row 4 was NOT moved, deliberately.** This proves the snapshot
|
||
*contains* a recoverable unit; it does not prove a restore puts data back into a running app. §10.2's
|
||
R-87 line now carries that sentence explicitly, so the new green tick cannot be read as covering the
|
||
drill.
|
||
|
||
---
|
||
|
||
## 11. Teardown — all three layers
|
||
|
||
| layer | created | after |
|
||
|---|---|---|
|
||
| PVE host `demo-hp` `/root` | 5 scripts + one 0600 password file | `ls \| grep` → nothing |
|
||
| guest 9201 `/root`, `/tmp` | 11 files | grep → nothing (two leftovers found on the first pass and removed) |
|
||
| container `/tmp` | env, sampler, log, run-flag, constructed tree | `ls -A /tmp` → **empty** |
|
||
|
||
**The off-site store** holds one extra `opengist` snapshot (`f32e1078`, the declared construction).
|
||
Nothing was deleted from it. Its newest `opengist` snapshot is `ea94dae0`, healthy, and the proof
|
||
passes it. **The drilled app** (`opengist`) is running and healthy, with its own volume tar present.
|
||
**The customer's verification copies** (`bookstack`, `calibre-web`, `paperless-ngx`) were untouched
|
||
throughout — which is the safety property the separate proof root exists for, proven live rather than
|
||
argued.
|
||
|
||
---
|
||
|
||
## 12. Register
|
||
|
||
| id | action |
|
||
|---|---|
|
||
| **R-87** | **CLOSED** — shipped + proven-live, then compressed into `CLOSED-ITEMS.md` |
|
||
| R-242 | unchanged — a golden carrying 0.231.0 is now owed |
|
||
| R-408, R-409 | unchanged and still open; both are referenced by this work and neither was fixed |
|
||
|
||
**Register size, counted from git rather than from memory: `OPEN-ITEMS.md` **172 → 171** rows (R-87 moved out); `CLOSED-ITEMS.md` **151 → 152**.** R-87 now appears exactly once, in `CLOSED-ITEMS.md`, and `closed_register_gate.py` confirms it is not in both.
|
||
No new rows were minted — every gap this session found is either fixed here or already has a row.
|
||
|
||
**`golden-currency` is RED and that is a DECLARED, EXPECTED debt:** controller v0.231.0 is released
|
||
and the newest golden carries 0.230.0. **The fleet is on 0.230.0. A golden carrying 0.231.0 is
|
||
OWED, and it is Viktor's call (R-242).** Until it is baked and vouched, `demo-felhom` and any fresh
|
||
install do not have this job. Both pushes of the `felhom.eu` repo used `--no-verify` for that reason
|
||
and it is declared here.
|
||
|
||
---
|
||
|
||
## 12b. CI, confirmed by run ID rather than assumed
|
||
|
||
| repo | run | sha | status |
|
||
|---|---|---|---|
|
||
| `felhom-controller` | 116 | `e43b5ec` | **success** |
|
||
| `felhom-controller` | 117 | `303129e` | **success** |
|
||
| `felhom-controller` | 118 | `79f853f` | **success** |
|
||
| `felhom.eu` | 287 | `1aeaa30` (hub v0.110.0) | **success** |
|
||
| `felhom.eu` | 288 | `7ee2592` (the closing docs commit) | **failure** |
|
||
|
||
**All three controller pushes are green**, and both went through the pre-push hook **without a
|
||
bypass**. Run 288 is red, and I verified the cause on the exact pushed tree rather than assuming it:
|
||
`repo_gates.py --fast` convicts **golden-currency and nothing else** — 12 of 13 gates OK. That is the
|
||
declared debt of §12, not a defect in this work, and it clears when a golden carries 0.231.0.
|
||
|
||
---
|
||
|
||
## 13. Scope: two things beyond the task's list, both deliberate
|
||
|
||
1. **`felhom.eu/hub/` was touched.** The task listed `documentation/` and `STATUS.md` only. Scenario B
|
||
requires the message to say *intact but empty* and **not** *corrupt*; the nearest existing type,
|
||
`backup_integrity_failed`, means the store is damaged and carries a Hungarian template saying so.
|
||
Reusing it would have shipped the wrong sentence. Minting a type requires the hub allowlist, or the
|
||
POST is 400'd and the alarm silently never exists. Reasoned in `CONTEXT.md` ruling 4.
|
||
2. **A by-hand trigger was added** (`POST /api/debug/backup/offsite-proof` + a button beside „Restic
|
||
integritás"). Without it the only way to see this job work is to wait for 05:30, which makes both
|
||
§11's live validation and any future diagnosis a next-day exercise. Same function as the scheduled
|
||
job — no second code path.
|
||
|
||
---
|
||
|
||
## Observations
|
||
|
||
1. **A STOPPED app still dumps its volume.** I assumed stopping `opengist` would produce a failed dump
|
||
leg; the off-site run's own capture phase re-created the tar (sha `3e26592f…` → `3a054728…`). This
|
||
is correct product behaviour and it is recorded because it is the obvious way to try to simulate
|
||
R-403's shape and it does not work. **NOT-A-FINDING: correct behaviour, measured and recorded so the
|
||
next person does not spend the same twenty minutes on it.**
|
||
2. **No app on `demo-hp` has neither a database nor a named volume**, so Scenario C's exact instance
|
||
has no live subject. **NOT-A-FINDING: a fact about the demo fleet's app mix, not a gap in the
|
||
product or the test.**
|
||
3. **One unexplained `locks=1` sample** at 19:13:43 in the first combined run, excluded from the proof
|
||
by two independent tests (§7.4). **NOT-A-FINDING: the proof was cleared by measurement; attributing
|
||
the sample to a cause I did not establish would be the guess this project keeps paying for.**
|
||
4. **`ResolveDockerVolumeNames` cannot be used from inside a recovery unit** — it derives the compose
|
||
project from the file's parent directory, which inside a unit is the literal string `compose`. This
|
||
is why the volume half is an existence check. **NOT-A-FINDING: a documented consequence of the
|
||
unit's layout, recorded in `REUSE.md` and `CONTEXT.md` where the next reader will meet it.**
|