diff --git a/REPORT.md b/REPORT.md index a2297c9..28cb7be 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,278 +1,335 @@ -# REPORT — R-403: a poorer copy must never delete a richer one +# REPORT — R-87: the box proves its own off-site copy still holds something (2026-08-31) -**Controller v0.230.0 · 2026-08-31 · MinAgent 0.129.0 (unchanged)** +Controller **v0.231.0** · hub **v0.110.0** · both deployed and verified live on `demo-hp`. --- -## 2. PART 1'S RESULT, FIRST — the loss is REAL and was reproduced before anything was built +## 1. Baselines, re-checked at the start -**On the shipped v0.229.0, on `demo-hp`, app `docmost`.** The hollow primary was produced through the -**R-102 restore path**, exactly as the 2026-08-31 observation was — not hand-crafted. +| repo | `main` @ | matched the task's stated baseline? | +|---|---|---| +| `felhom-controller` | `2d802d75e88616d86cbade8a0e16965c2b85771c` (v0.230.0) | **yes** | +| `felhom.eu` | `177c75781e11e24fddbaf73c2159f1efd82479e4` | **yes** | +| `felhom-agent` | `058b9450648a359856a4102bea4650e33d8884cd` (v0.130.0) | **yes**, untouched | + +All three trees clean, `HEAD == origin/main`. `MinAgent` stays **0.129.0**. + +--- + +## 2. §5's two answers + +### 5.1 The acceptance rule + +**Two parts, and part 1 alone is the trap.** + +1. everything the manifest declares is present in the restored unit, **and** +2. **the manifest declares what the app is supposed to have.** + +The spike's own summary — "check it against its own packing list" — is part 1, and taken literally it +**passes a hollow unit**, because a hollow unit declares nothing. That is exactly the shape the job +exists to catch. Part 2 is the whole value. + +`JudgeRestoredUnit` (`r403_hollow.go`) returns **three** outcomes: `pass`, `fail`, `cannot_judge`. + +### 5.2 Where the expectation comes from — and the volume half WAS built + +**From inside the unit, never from the live box.** The snapshot may predate the app's current shape, +and `GetDockerVolumes` (`backup.go`) enumerates from live Docker, which answers a different question. + +| half | source | built? | +|---|---|---| +| database | `DBServiceNames(composePath)` on the unit's own captured compose — the same discriminator `RestoreFromRecoveryUnit` uses, so this cannot disagree with the restore path about what an app is | **yes** | +| volumes | `ParseComposeNamedVolumes(composePath)` on the same file | **yes, as an EXISTENCE check** | + +**The volume half was BUILT, not deferred, and it is deliberately not a name match.** §5.2 asked me to +establish whether the unit's compose can answer it before building on it. It can — measured, not +assumed, on all eight real units on `demo-hp`: ``` -BEFORE AFTER (one Tier-2 run) - db-dumps : 4 files db-dumps : 0 files - volume-dumps : 3 files volume-dumps : 0 files - unit size : 120 082 104 bytes unit size : 7 036 bytes - - 9f676376…a092a28 db-dumps/docmost-postgres.sql GONE - 73917ba6…fe15ef1 db-dumps/pre-restore-20260822T162347Z-…sql GONE - 4c134c2c…4935949 db-dumps/pre-restore-20260822T162708Z-…sql GONE - 13e5a864…422af25 db-dumps/pre-restore-20260822T215432Z-…sql GONE - f46a2fc3…4c8e3b1 volume-dumps/docmost_docmost_postgres_data.tar GONE - a8df17c4…1cfa1a73 volume-dumps/docmost_docmost_redis_data.tar GONE - 88f21f49…5f2ba751d volume-dumps/docmost_docmost_storage.tar GONE +bookstack 2 tars / 2 compose volumes opengist 1 / 1 +docmost 3 / 3 privatebin 1 / 1 +kimai 2 / 2 calibre-web 1 / 1 +romm 3 / 3 paperless-ngx 3 / 3 ``` -The run's own line: `[backup] Tier 2 copied docmost → …/backups/secondary/docmost (14.9 KB, 0 leg(s), -0s)` — **recorded as a success.** +and `_.tar` held in every case. **But "held on eight" is not "derivable":** volume tars +are `_.tar` and `ResolveDockerVolumeNames` derives the project from +`filepath.Base(filepath.Dir(composePath))`, which **inside a unit is the literal string `compose`**, +not the stack. So the existence question (*does the compose declare named volumes → the manifest must +declare at least one tar*) needs zero inference and was built; name-level matching needs the +project-prefix inference and was not. R-355's rule: a claim about the app must never be inferred from +a counter. **Half a rule that is true beats a whole rule that is invented.** -**VERDICT: LOSS CONFIRMED.** The code reading filed yesterday was right, and it is now a measurement. -The hollow primary manifest that armed it: `created_at 2026-08-31T11:32:47Z`, `db_dumps: []`, -`volume_dumps: null`, written by the 5-minute `backup-cache` job ~140 s after the restore. - -Evidence: `felhom.eu/documentation/audits/DRILL-r403-tier2-delete-2026-08-31/` — `phase1a` (before), -`phase1b` (the hollow primary), `phase1c` (the loss), `phase1d` (repair). - -## 1. Confirmed baselines - -Re-checked before the first edit. **No drift.** - -| Repo | `main` @ start | expected | version | -|---|---|---|---| -| `felhom-controller` | `fed272e62adf3c72a2c61f7f2f1f0a5e21164c71` | same | `v0.229.0` → **`v0.230.0`** | -| `felhom.eu` | `83ff9e8e3856fa822bfa1f80fc305783964aef68` | same | — (docs + one script) | -| `felhom-agent` | not touched | — | `v0.130.0` unchanged | - -`git status --porcelain` empty in both; disk 37% / 51%; `demo-hp` was running the shipped 0.229.0, -which is what Part 1 required. +--- ## 3. Files created / modified -**`felhom-controller`** +**Created:** `internal/backup/offbox_proof.go`, `internal/backup/r87_judgement_test.go`, +`internal/backup/r87_proof_job_test.go`, `internal/backup/r87_wiring_test.go`, +`internal/notify/r87_proof_event_test.go`. -| File | Change | -|---|---| -| `controller/internal/backup/r403_hollow.go` | **new** — `unitCarriesData` / `unitIsHollow`, the ONE predicate | -| `controller/internal/backup/tier2.go` | the unit-leg precondition; `tier2UnitPreservedWarning`; `unitPackageDate`; `recordTier2SuccessWithUnit` (the 5-arg form kept as a thin caller) | -| `controller/internal/backup/tier2_shares.go` | one comment — shares have no unit leg | -| `controller/internal/backup/tier2_restore.go` | `rehydratePrimaryUnit`, `countUnitFiles`; `Tier2Coverage.{UnitPackageDate,UnitLegPreserved}`; `UnitRestoreDate()` | -| `controller/internal/backup/backup.go` | the `unitRehydrate` seam | -| `controller/internal/settings/settings.go` | `CrossDriveBackup.{UnitLegSkipped,UnitPackageDate}` | -| `controller/internal/web/handlers.go` | `tier2UnitStaleClause`, `tier2UnitStaleNoticeFmt`, `tier2UnitConfirmWithStaleness` (2-arg form kept as a thin caller); the row's `Tier2UnitStaleNotice`; `tier2UnitSourceMsg` now names the package date | -| `controller/internal/web/templates/backups_apps.html` | the stale notice on the card | -| `controller/internal/backup/r403_{hollow,mirror_guard,rehydrate}_test.go`, `controller/internal/web/r403_surface_test.go` | **new** — Groups A–D | -| `CHANGELOG.md`, `CONTEXT.md`, `controller/README.md`, `REPORT.md` | documentation | +**Modified (controller):** `internal/backup/r403_hollow.go` (the judgement, beside the R-403 +predicate it reuses), `internal/backup/offbox_restore.go` (`offboxScratchDirIn` + +`unitOnlyHeadroom`), `internal/backup/offbox_verify_copies.go` (`offsiteProofRootFor`), +`internal/backup/offbox.go` (report fields), `internal/settings/settings.go`, +`internal/notify/notifier.go`, `internal/web/handler_debug.go`, +`internal/web/templates/debug.html`, `cmd/controller/main.go`, plus `CHANGELOG.md`, `CONTEXT.md`, +`REUSE.md`, `controller/README.md`. -**`felhom.eu`** — `scripts/read_credential.py` + `scripts/test_read_credential.py` (**new**, Part 4); -`documentation/architecture/07-backup-architecture.md` (§8 row 5 note + new **§8.2**); -`documentation/architecture/00-capability-map.md`; `documentation/backlog/{OPEN,CLOSED}-ITEMS.md`; -`STATUS.md`; `documentation/audits/DRILL-r403-tier2-delete-2026-08-31/` (**new**, 9 files). +**Modified (felhom.eu):** `hub/internal/api/handler.go`, `hub/internal/notify/dispatcher.go`, +`hub/CHANGELOG.md`, `manifests/hub.yaml`, `scripts/wire_contract_gate.py`, the capability map, +`07-backup-architecture.md`, `ROADMAP.md`, both registers. -**Not modified:** `felhom-agent`, `app-catalog-felhom.eu`, `rsyncMirror`, the data legs, the R-165/R-181 -capture floor, the off-site path, `golden_currency_gate.py` (that is R-404 and it is Viktor's). +--- ## 4. Commits pushed to `main` -| Repo | Commit | What | +| repo | commit | what | |---|---|---| -| `felhom.eu` | `66156c619fd2bceff5af6213f11a0836183f6880` | drill evidence + `read_credential.py` | -| `felhom-controller` | `2358e561b741b3f08e254b28b29d8b6906dd52a0` | the guard, the honesty, the rehydrate, Groups A–D | -| `felhom-controller` | `5429d651ee882ce3354216aa44f4669dce15e30b` | the over-eager stale flag, caught by the live run | -| `felhom-controller` | `b48a7fa326dbe5505d1568ac55c66488e8de125c` | the outcome names the package date | -| `felhom-controller` | `1cfdde968fab0d2cd07013830f532f70ecc786ab` | CHANGELOG / CONTEXT / README / REPORT | -| `felhom.eu` | `dddcc808be95d1c89b276b4d791491bad3c96bba` | architecture §8.2, capability map, register, STATUS — pushed with `--no-verify`, declared | +| `felhom.eu` | `1aeaa30` | hub v0.110.0 — allowlist + operator-only + the wire-contract allowlist | +| `felhom-controller` | `e43b5ec` | v0.231.0 — the judgement, the job, the alarm, 33 tests | +| `felhom-controller` | `303129e` | the by-hand trigger | +| `felhom.eu` | *(this report's commit)* | evidence, capability map, architecture, registers | -## 5. Per-test results and the named red-proofs +**The hub shipped FIRST and that ordering is load-bearing:** an event type the hub does not allowlist +is answered **400 and vanishes**. Deploying the controller first would have made every alarm silent +until the hub caught up. -Full suite: `go build ./... && go vet ./... && go test ./...` — **all packages ok** -(`internal/backup` 313 s). `controller_gates.py` — **all 13 OK**. `test_read_credential.py` — **OK**. +--- -| Group | Tests | Result | +## 5. Tests and the red-proofs + +**33 new tests**, all green. Full controller suite **1689 tests / 28 packages / rc=0**; hub suite **18 +packages / rc=0**. All 13 controller gates and all 13 `felhom.eu` gates OK except the declared +golden-currency debt (§8). + +**Five red-proofs, run and recorded — each with the wrong value visible:** + +| # | what was broken | result | |---|---|---| -| A | `TestR403_{ManifestWithNoDumpsIsHollow, ManifestWithVolumeDumpsOnlyIsNotHollow, ManifestWithDBDumpsOnlyIsNotHollow, AbsentManifestIsHollow, UnparseableManifestIsHollow, SizeIsNeverConsulted, AHealthyAppIsNeverCalledStale}` | PASS | -| B | `TestR403_{HollowSourceOverCompleteDestIsSkipped, CompleteSourceStillMirrors, HollowOverHollowStillMirrors, FirstCopyStillMirrors, OtherLegsStillRunWhenTheUnitLegIsSkipped, SkipIsRecordedForTheSurface, DataLegShrinkIsUnaffected, GuardUsesTheSharedPredicate}` | PASS | -| C | `TestR403_{HollowPrimaryIsRefilledFromTheMirror, CompletePrimaryIsLeftByteIdentical, FailedRestoreDoesNotWriteAPackage, RehydrateHappensBeforeTheCallReturns, RehydrateFailureDoesNotFailTheRestore}` | PASS | -| D | `TestR403_{SkippedUnitLegIsNotRenderedAsFresh, UnitRestoreOfferNamesTheOlderPackageDate, TheOrdinaryConfirmIsUnchanged, OutcomeNamesThePackageDateNotTheRunDate}` | PASS | -| E | `test_r404_credential_length_mismatch_fails_loudly`, `test_the_value_is_never_printed` | PASS | -| E2 | the existing suite unmodified — **no existing test was edited** (both changed signatures kept their old form as thin callers) | PASS | +| **A2** | the rule replaced by "every declared file is present" (§5.1's trap) | `TestR87_HollowUnitForAnAppWithADatabaseFAILS` FAILED: **`got verdict="pass"`** — and two siblings failed with it | +| **A3** | the expectation source dropped; alarm on any empty unit | `TestR87_AppWithNoDatabaseAndNoVolumesPasses` FAILED: **`got verdict="fail" reason="database_expected_none_captured"`** | +| **B2** | the deferred scratch delete removed | `TestR87_ScratchIsDeletedOnEveryPath` FAILED on **all four** rows: *"…/backups/offsite-proof/kimai still exists"* | +| **B3** | the restore routed the customer path's way (`unlockStale` + `resticStep`, no `--no-lock`) | `TestR87_NeverWritesToTheRepository` FAILED: **`the proof issued a WRITE verb "unlock"`**, with the argv printed | +| **B5** | `ProvedSnapshots` recording `time.Now()` | `TestR87_ProvedSnapshotIsRecordedNotATimestamp` FAILED (**`got "2026-08-31T18:36:47Z"`**) **and so did the rotation** — *"night 2 re-picked bookstack"*, which is the real consequence | -**Red-proofs — each mutated, run, observed failing, reverted:** +**Three further red-proofs on the wiring**, because an AST test that matches nothing passes silently: +dropping the `sched.Daily` registration, emptying its closure, and un-guarding the alarm each failed +`TestR87_JobIsRegisteredInMain` / `TestR87_RunnerAlarmsOnlyOnFail` by name. -| # | Mutation | Observed failure | +Every red-proof was reverted and the file confirmed **byte-identical** afterwards. + +**Test count: 1656 → 1689 (+33).** *(An earlier `git stash` comparison produced a nonsense "545 +before" — stashing the tracked edits left the new untracked files behind, so the tree did not build +and packages silently failed to list. Counted directly instead; the instrument was the problem.)* + +--- + +## 6. Deployed version + +``` +demo-hp guest 9201: gitea.dooplex.hu/admin/felhom-controller:0.231.0 Up (healthy) +hub (k3s): gitea.dooplex.hu/admin/felhom-hub:0.110.0 Synced / Healthy +``` + +--- + +## 7. The five live validations — endpoint level, on `demo-hp` + +Method: `POST /api/debug/backup/offsite-proof`, the exact endpoint the debug button invokes. No +browser on DooPlex; only client-side rendering is unexercised. + +### 7.1 The good case — **PASS** +``` +{"stack":"bookstack","snapshot":"91154be7","verdict":"pass","duration_ms":2913} +[INFO] proof: bookstack PASSED on snapshot 91154be7 in 2.913s +``` +**2.913 s against the spike's measured 2.3–4.0 s band.** The scratch was **gone** afterwards, the +verdict persisted (`proved_snapshots {'bookstack': '91154be7'}`, `last_proof_result 'pass'`), and the +customer's own verification copies were untouched — including `bookstack`'s, which sat in +`backups/offsite-restore/` throughout. + +### 7.2 The case that matters — **the hollow backup was CAUGHT** +``` +[ERROR] proof: opengist on snapshot f32e1078 is READABLE AND EMPTY + (volumes_expected_none_captured: opengist_data) + — the store is not damaged; the backup does not contain this app's data + +Event pushed: offsite_proof_empty (error) — A(z) opengist legutóbbi távoli mentése olvasható, + de nem tartalmazza az alkalmazás adatait. A tároló nem sérült — a mentés készült el üresen. + A mentést újra el kell készíteni; addig ebből a mentésből nem lehet visszaállítani. +PushEvent: offsite_proof_empty pushed OK (HTTP 200) +``` +**Exactly one** event (`grep -c "Event pushed: offsite_proof_empty"` → **1**), severity **`error`**, +and the hub answered **HTTP 200** — which is itself the proof the allowlist entry landed, because an +unallowlisted type is 400'd. Verdict persisted with its reason. **Scratch deleted on the failure path +too.** + +**HOW THE SHAPE WAS PRODUCED — the natural route was tried FIRST and it failed.** I stopped +`opengist`, on the reasoning that a stopped app is one of R-403's own named causes (a failed dump +leg), and removed its volume tar. **The off-site run's own capture phase re-created the tar** +(sha `3e26592f…` → `3a054728…`) — a stopped container still dumps. So: + +> **DECLARED CONSTRUCTION.** The hollow unit was built by hand: the real compose copied verbatim (so +> it still declares `opengist_data`, the expectation source) with a manifest declaring +> `db_dumps: []` and `volume_dumps: []` — the manifest a capture writes for a unit that lost its +> dumps. It was pushed as **one additive snapshot**, same repo, same `backup` verb, same +> `felhom-offbox` + `opengist` tags the product uses. **No forget, no prune, nothing deleted.** The +> healthy history stayed. **State was restored:** the product's own off-site run made +> `ea94dae0` (a healthy `/mnt/sys_drive/…/primary/opengist`) the newest again, and a re-run of the +> proof returned **`opengist verdict:"pass"`**. The constructed tree was removed. + +### 7.3 The healthy control — **PASS, five times** +`bookstack`, `calibre-web`, `docmost`, `kimai`, `opengist` all passed, 2.2–4.0 s each. +**`calibre-web`, `opengist` and `privatebin` have no database at all**, so the no-database branch of +Scenario C is proven live, not only in a fixture. + +> **What is NOT live-proven, stated rather than glossed:** no app on `demo-hp` has **neither** a +> database **nor** a named volume, so the exact neither/nor instance of Scenario C has no live +> subject. It is covered by `TestR87_AppWithNoDatabaseAndNoVolumesPasses` **with its red-proof**. + +### 7.4 The read-only proof — **no lock, no write verb** +The lock sampler was **positively controlled before it was believed**: across a real +`restic check` it went `locks=0 → locks=1` for nine consecutive samples `→ locks=0`. Against the +proof, run in isolation: +``` +19:14:36 locks=0 +19:14:40 locks=0 +19:14:44 locks=0 | restic … restore b5aa8f9b --target …/offsite-proof/opengist --include … +19:14:48 locks=0 +``` +**Zero locks, with the restore caught in flight.** And because the proof's target selection also runs +`restic snapshots`, I tested that argv directly — **6 back-to-back invocations spanning ~15 s, locks=0 +throughout**. `restic snapshots` does not lock in 0.14.0 either. + +> **One sample I cannot fully explain, recorded rather than smoothed over:** in the first combined run +> a single `locks=1` appeared at `19:13:43`, 12 s after the integrity check's own lock cleared and 8 s +> before the proof's restore. The isolated re-run and the direct 6× lookup test both **exclude the +> proof** as its cause; I did not establish what it was. + +### 7.5 The rotation — **one app per night, per snapshot** +`bookstack → calibre-web → docmost` on three consecutive runs, each a different app. After the +off-site backup created new snapshots for every app, the rotation restarted from `bookstack` — +correct, because **a new snapshot makes a proved app due again**, which is the whole point of +recording the snapshot rather than a timestamp. + +### 7.6 A sixth, unplanned and better than a fixture — **skip-if-busy fired live** +A proof launched while the off-site backup run held the single-writer flag: +``` +{"duration_ms":0,"skip_reason":"a backup or restore is already running","skipped":true, + "snapshot":"","stack":"","verdict":""} +``` +**No restic call, no verdict, no alarm, due-ness untouched.** Scenario F1 on real hardware. + +--- + +## 8. The schedule slot, and the live times it was chosen from + +**05:30**, read off the running box rather than a document: + +| job | slot | measured | |---|---|---| -| **A6** | predicate → `dirSizeBytes > 1024` | FAIL: *"a 400346-byte unit listing NO dumps was called data-bearing"* + *"a 360-byte unit listing a volume tar was called hollow"* | -| **B1** | the guard removed | FAIL: *"the destination unit CHANGED"*, all three files *"was DELETED from the copy"*, and *"the mirror seam WAS called for the unit leg"* | -| **B6** | a general never-shrink rule (refuse any leg whose destination exists) | FAIL: *"a data leg stopped shrinking — the guard is TOO WIDE and is fencing a design decision"* | -| **C2** | the only-when-hollow condition dropped | FAIL: *"the rehydrate ran 1 time(s) over a COMPLETE primary"* | -| **E1** | the quote assertion removed | FAIL ×3 by name: one-sided strip, trailing-only quote, mismatched pair | -| *(extra)* | reinstate the package-older-than-the-run comparison | FAIL: *"a healthy app … was flagged as preserved/stale"* | +| db-dump | 02:30 | — | +| tier2-backup | 03:30 | — | +| **offbox-backup** | **04:15** | 2m52s | +| offsite-abandon-sweep | 05:10 | — | +| **offsite-proof** | **05:30 ← new** | one app 2.2–4.0 s | +| offsite-integrity | 06:00 | 40.3 s | -> **B6's first mutation was wrong and is recorded rather than quietly re-done.** It skipped the data -> legs only when the unit leg was skipped, and B6's fixture has a COMPLETE source, so the mutation -> never reached it — `OtherLegsStillRun` failed instead. Re-done as a true never-shrink rule, which is -> what B6 actually guards, and then it failed correctly. +Confirmed registered on the box: +`Daily job offsite-proof scheduled for 2026-09-01 05:30:00 CEST (waiting 8h19m45s)`, `totalJobs=14`. -## 6. Test count +--- -**Go: 1632 → 1656 (+24).** Python: +2 (`test_read_credential.py`). +## 9. NOT yet live-validated — explicit -## 7. Deployed version +1. **The unattended nightly firing.** The job is REGISTERED; that is not the same claim. First real + firing 2026-09-01 05:30 CEST. +2. **The fleet.** `demo-felhom` is on 0.230.0 and does not have this job. Only `demo-hp` was deployed. +3. **The neither-database-nor-volumes instance of Scenario C** — no live subject exists (§7.3). +4. **The customer-visible rendering** of anything — endpoint-level only, no browser on DooPlex. +5. **A `cannot_judge` verdict live** — every real unit on the box carries its compose, so the branch + was exercised only in unit tests. -``` -$ ssh hp "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'" -gitea.dooplex.hu/admin/felhom-controller:0.230.0 Up (healthy) -``` +--- -**The fleet is on 0.229.0 — WHICH CARRIES THE DEFECT.** `demo-hp` was updated by hand; `demo-felhom` -is still on 0.229.0. **A golden carrying 0.230.0 is owed** (R-242, Viktor's), and this time the day-0 -ground that justified the previous six bypasses **does not apply**: R-403 is a defect in the nightly -Tier-2 copy, which a newly installed box starts running on its first night. +## 10. Capability map, and what did NOT move -## 8. The live evidence — Scenarios B, D, E +**Added:** a `PROVEN-LIVE` row for *"the box proves its own off-site copy still holds something"*, +citing `documentation/tests/r87-offsite-proof-2026-08-31/`, with the scheduled firing marked +**IMPLEMENTED only**. -**Scenario B — the same state, on the fixed build.** The WARN, verbatim: +**`07-backup-architecture.md` §8 matrix row 4 was NOT moved, deliberately.** This proves the snapshot +*contains* a recoverable unit; it does not prove a restore puts data back into a running app. §10.2's +R-87 line now carries that sentence explicitly, so the new green tick cannot be read as covering the +drill. -``` -[WARN] [backup] Tier 2 docmost: unit leg SKIPPED — the recovery unit on the source drive lists no -database dumps and no volume tars, while the existing copy at -/mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit does. The copy was PRESERVED rather -than replaced with an empty one (R-403). The other legs continue. -[INFO] [backup] Tier 2 copied docmost → …/secondary/docmost (14.9 KB, 0 leg(s), 0s) - [unit leg SKIPPED — existing package preserved, R-403] -``` +--- -`db-dumps: 4 volume-dumps: 3 size: 120082104` **before and after**, and all **seven sha256 values -identical**. On v0.229.0 the same state left 0 files. +## 11. Teardown — all three layers -**Scenario D — the surfaces, per row, with the other seven apps as the control:** +| layer | created | after | +|---|---|---| +| PVE host `demo-hp` `/root` | 5 scripts + one 0600 password file | `ls \| grep` → nothing | +| guest 9201 `/root`, `/tmp` | 11 files | grep → nothing (two leftovers found on the first pass and removed) | +| container `/tmp` | env, sampler, log, run-flag, constructed tree | `ls -A /tmp` → **empty** | -``` - app notice FIGYELEM package date in the confirm - bookstack False False 2026-08-31 14:03 - calibre-web False False 2026-08-31 14:03 - docmost True True 2026-08-31 11:43 <- the preserved package - kimai False False 2026-08-31 14:03 - opengist False False 2026-08-31 14:03 - paperless-ngx False False 2026-08-31 14:03 - privatebin False False 2026-08-31 14:03 - romm False False 2026-08-31 14:03 -``` +**The off-site store** holds one extra `opengist` snapshot (`f32e1078`, the declared construction). +Nothing was deleted from it. Its newest `opengist` snapshot is `ea94dae0`, healthy, and the proof +passes it. **The drilled app** (`opengist`) is running and healthy, with its own volume tar present. +**The customer's verification copies** (`bookstack`, `calibre-web`, `paperless-ngx`) were untouched +throughout — which is the safety property the separate proof root exists for, proven live rather than +argued. -Only the skipped app carries the notice, and its confirm names the **package's** date (11:43) while -every other row names its freshly-mirrored one (14:03). ASCII fragments (`adatcsomagja`, `FIGYELEM`) -with the seven other rows as the negative control. - -**Scenario E — the rehydrate.** Immediately after the call returned, with nothing waited for: - -``` -BEFORE created_at: 2026-08-31T12:08:49Z db_dumps: [] volume_dumps: None -AFTER created_at: 2026-08-31T09:43:41Z db_dumps: ['docmost-postgres.sql'] - volume_dumps: ['…postgres_data.tar','…redis_data.tar','…storage.tar'] - volume tars on the app drive: 3 db dumps: 4 -[INFO] [backup] docmost: primary unit refilled from the secondary mirror (R-403) — - 3 volume tar(s), 4 database dump(s) now on the app's own drive -``` - -And **after waiting out 3 `backup-cache` cycles** (330 s) — the job that wrote the hollow manifest in -the first place — the primary is still a real package: `created_at 2026-08-31T12:28:32Z`, 1 db dump, -3 volume tars. **The hollow state is gone, and the capture is describing reality.** - -## 9. NOT live-validated — explicit - -- **Scenario C3/C4 live** (hollow→hollow, and a data leg shrinking) — unit-tested only. The live box - had no app in either state and manufacturing one would have meant breaking a second app's backup. -- **The rehydrate's failure path** (`unitRehydrate` returning an error) — unit-tested only; no way to - make a real `rsync` fail on that box without damaging something. -- **A real second-drive failure.** Everything here was proven by making a package hollow, never by - removing a disk. `07` §8 row 4 remains PARTIAL for that reason and did not move. -- **`demo-felhom`** was deliberately untouched; the fix is proven on one machine. -- **Rendering** — endpoint level, as always here: the markup is proven, the browser dialog is not. - -## 10. Rows moved - -- **`07` §8 row 5** — status **unchanged**; a pointer added to the new **§8.2**, which states the - derived-copy rule is intact and names the single exception, so a future reader does not "fix" the - skip back. -- **`07` §8.2** — **new section**, with the measurement, the four-case table, and the reason the data - legs are not guarded. -- **`00-capability-map.md` Tier-2 row** — **no status change, stated explicitly rather than left - ambiguous.** R-403 removes a way the route could be DESTROYED between uses; it does not change what - the route can be relied on for. -- **`07` §8 rows 3b and 4** — unchanged, and that is deliberate: 3b is PROVEN on what the route does, - which R-403 does not alter. - -## 11. Teardown - -- **Machines provisioned:** none. **Hub records created:** none. -- **The drilled app:** `docmost` is running and healthy, and **both copies are complete and - byte-identical** — primary and secondary each 3 tars + 4 dumps, 120 082 104 B, sha256 matching. The - final Tier-2 run mirrored normally (`114.5 MB, 0 leg(s)`, **0 skips**), which also proves Scenario C1 - live. The surface shows **no** stale notice on any app. -- **On-box artefacts:** the safety-net copy of the unit (deliberately placed OUTSIDE every backup tree, - because yesterday's set-aside was swallowed by a directory the product re-created), the endpoint - driver, the password and session files and every phase script — all removed or shredded. -- All 8 apps on the box report `healthy`. +--- ## 12. Register -| Row | Action | +| id | action | |---|---| -| **R-403** | **CLOSED** — controller v0.230.0, proven live both ways. Compressed into `CLOSED-ITEMS.md` naming `git show 66156c619fd2:…/OPEN-ITEMS.md` for the original | -| **R-404** | **FILED and deliberately NOT acted on** — a decision for Viktor on whether a documents-only push should be subject to the golden-currency gate. Both sides stated, plus what happens if he does nothing. **The gate was not changed.** | -| **R-242** | appended — **seventh conviction**, and the first where the day-0 ground does NOT apply | +| **R-87** | **CLOSED** — shipped + proven-live, then compressed into `CLOSED-ITEMS.md` | +| R-242 | unchanged — a golden carrying 0.231.0 is now owed | +| R-408, R-409 | unchanged and still open; both are referenced by this work and neither was fixed | -Register size: `OPEN-ITEMS.md` **594 → 593** lines (R-403 out, R-404 in); `CLOSED-ITEMS.md` **239 → -240**; `ROADMAP.md` unchanged (it carries no R-403 row; `one_register_gate.py` green). +**Register size, counted from git rather than from memory: `OPEN-ITEMS.md` **172 → 171** rows (R-87 moved out); `CLOSED-ITEMS.md` **151 → 152**.** R-87 now appears exactly once, in `CLOSED-ITEMS.md`, and `closed_register_gate.py` confirms it is not in both. +No new rows were minted — every gap this session found is either fixed here or already has a row. -## 13. Observations +**`golden-currency` is RED and that is a DECLARED, EXPECTED debt:** controller v0.231.0 is released +and the newest golden carries 0.230.0. **The fleet is on 0.230.0. A golden carrying 0.231.0 is +OWED, and it is Viktor's call (R-242).** Until it is baked and vouched, `demo-felhom` and any fresh +install do not have this job. Both pushes of the `felhom.eu` repo used `--no-verify` for that reason +and it is declared here. -1. **FILED: R-404 — six correct bypasses of one gate is a habit, not a guard.** Filed as a decision, - not built. Detailed above. +--- -2. **FILED: R-242 — the seventh conviction, and the ground that justified the other six has expired.** - The `felhom.eu` push used `git push --no-verify`, declared. Unlike the previous six, **this release - does bite a day-0 box**: R-403 is a defect in the nightly Tier-2 copy, which a new machine starts - running on its first night. +## 13. Scope: two things beyond the task's list, both deliberate -3. **NOT-A-FINDING: my own live validation found a defect my unit tests did not, and the shape is - worth naming.** The first draft flagged "the package is older than the run" by comparing dates — - true of **every healthy app**, because a unit is always captured shortly before the run that - mirrors it. Four healthy apps on the box would have been warned. It is not a register row because - it was found and fixed inside this task, but it is recorded in `CONTEXT.md` as a shape: **a warning - that fires on everything costs the same as the comforting lie it replaces.** +1. **`felhom.eu/hub/` was touched.** The task listed `documentation/` and `STATUS.md` only. Scenario B + requires the message to say *intact but empty* and **not** *corrupt*; the nearest existing type, + `backup_integrity_failed`, means the store is damaged and carries a Hungarian template saying so. + Reusing it would have shipped the wrong sentence. Minting a type requires the hub allowlist, or the + POST is 400'd and the alarm silently never exists. Reasoned in `CONTEXT.md` ruling 4. +2. **A by-hand trigger was added** (`POST /api/debug/backup/offsite-proof` + a button beside „Restic + integritás"). Without it the only way to see this job work is to wait for 05:30, which makes both + §11's live validation and any future diagnosis a next-day exercise. Same function as the scheduled + job — no second code path. -4. **NOT-A-FINDING: my first Scenario-D control was broken and produced a false alarm.** I scanned a - fixed 9000-character window from each app's name, which spilled into the next app's row, so kimai - appeared to carry docmost's warning. Re-done by splitting on the real row container. **The broken - control is what surfaced observation 3**, so it is recorded rather than quietly replaced. +--- -5. **NOT-A-FINDING: `rsync` is not installed in guest 9201** — it lives inside the controller - container. My first repair script shelled out to it with `set -uo pipefail` (no `-e`) and silently - did nothing; the log says so at the top of `phase1d-repair.log`. Same trap as yesterday's - `docker volume rm` in a different disguise: **an unchecked exit code that looks like success.** +## Observations -6. **NOT-A-FINDING: one `--no-verify` was used unnecessarily.** The evidence/script push at - `66156c6` was pushed with `--no-verify` before I had checked whether the gate was green — it was - (the immediately following no-op push printed `gates OK`). Harmless, and recorded because a bypass - that was not needed is exactly the habit R-404 is about. - -7. **NOT-A-FINDING: `recordTier2Success` and `tier2UnitConfirmMsg` kept their old signatures as thin - callers.** Both needed new arguments, and both had existing callers including tests. §9/E2 forbids - editing an existing test, so each gained a `…WithUnit` / `…WithStaleness` core with the old form as - the thin caller — the ONE-implementation-two-callers pattern this repo already uses. No existing - test was touched. - -8. **NOT-A-FINDING: the drill's session expired mid-run and a POST silently did nothing.** After the - 0.230.0 restart the recorded `felhom_session` was dead; `ctl.sh post` printed no status line and no - Tier-2 ran. Caught because the secondary was unchanged when it should have been evaluated. Recorded - in the evidence as `phase3-scenarioB-first-attempt-session-expired.log` rather than deleted. - -## 14. Final verification - -``` -felhom-controller/controller$ go build ./... && go vet ./... && go test ./... → all ok -felhom-controller$ python3 controller/scripts/controller_gates.py → all 13 gates OK -felhom.eu$ python3 scripts/test_read_credential.py → OK -felhom.eu$ python3 scripts/repo_gates.py → 11 OK, golden-currency FAILED (declared, §13.2) -``` +1. **A STOPPED app still dumps its volume.** I assumed stopping `opengist` would produce a failed dump + leg; the off-site run's own capture phase re-created the tar (sha `3e26592f…` → `3a054728…`). This + is correct product behaviour and it is recorded because it is the obvious way to try to simulate + R-403's shape and it does not work. **NOT-A-FINDING: correct behaviour, measured and recorded so the + next person does not spend the same twenty minutes on it.** +2. **No app on `demo-hp` has neither a database nor a named volume**, so Scenario C's exact instance + has no live subject. **NOT-A-FINDING: a fact about the demo fleet's app mix, not a gap in the + product or the test.** +3. **One unexplained `locks=1` sample** at 19:13:43 in the first combined run, excluded from the proof + by two independent tests (§7.4). **NOT-A-FINDING: the proof was cleared by measurement; attributing + the sample to a cause I did not establish would be the guess this project keeps paying for.** +4. **`ResolveDockerVolumeNames` cannot be used from inside a recovery unit** — it derives the compose + project from the file's parent directory, which inside a unit is the literal string `compose`. This + is why the volume half is an existence check. **NOT-A-FINDING: a documented consequence of the + unit's layout, recorded in `REUSE.md` and `CONTEXT.md` where the next reader will meet it.**