diff --git a/CHANGELOG.md b/CHANGELOG.md index dde29f8..32b0197 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,97 @@ +## v0.230.0 — a poorer copy must never delete a richer one (2026-08-31, R-403) +**MinAgent: 0.129.0** (unchanged) + +### The loss was MEASURED first, then fixed + +R-403 was filed yesterday from a code reading with the dangerous half explicitly recorded as +**unverified**. It was run before anything was built, on the shipped **v0.229.0**, on `demo-hp`: + +``` +BEFORE secondary db-dumps: 4 volume-dumps: 3 size: 120082104 bytes + (a hollow primary unit, produced through the R-102 restore path exactly as on 2026-08-31) +RUN [backup] Tier 2 copied docmost → …/backups/secondary/docmost (14.9 KB, 0 leg(s), 0s) +AFTER secondary db-dumps: 0 volume-dumps: 0 size: 7036 bytes +``` + +**Four database dumps and three volume tars — the customer's last surviving package — deleted in one +nightly run, which recorded itself a SUCCESS.** Evidence: +`felhom.eu/documentation/audits/DRILL-r403-tier2-delete-2026-08-31/`. + +The mechanism was three individually-correct lines: `RunTier2` guarded the unit leg with `os.Stat` +alone (*does the folder exist*), `rsyncMirror` is `rsync -a --delete`, and nothing between them +compared source to destination. **An empty recovery unit is a folder that exists.** + +### The guard + +- **One predicate, `unitCarriesData` / `unitIsHollow`** (`internal/backup/r403_hollow.go`). It **asks + the MANIFEST and never the byte size**: a unit with a big compose tree and no dumps is dangerous, a + tiny unit for a tiny app is fine. Fail-closed on an absent or unparseable manifest. + `TestR403_SizeIsNeverConsulted` is the guard that keeps `dirSizeBytes` out. +- **`RunTier2` skips the unit leg** when the source is hollow **and** the destination is not. The + other legs still run, the run is **not** failed, and the skip is recorded **for the surface** + (`CrossDriveBackup.UnitLegSkipped` + `UnitPackageDate`) as well as logged loudly. +- **`--delete` STAYS and shrinking stays legal.** `07-backup-architecture.md` §8 row 5's derived-copy + rule ("Migration = rebuild, not preserve") is unchanged, and `tier2.go`'s own header records that a + classified app's copy legitimately shrinks as `export` drops out. The fence is exactly one shape. + `TestR403_DataLegShrinkIsUnaffected` is the guard on the guard. + +### The honesty — a preserved copy must not read as a fresh one + +A preserved package is older than the run that preserved it. The per-app card carries a notice, and +the unit-restore confirm names the **package's** date — read from the mirrored manifest's own +`created_at`, not from the status record — plus a clause saying why it is older. **The outcome +sentence names the same date**, because §2.3's rule is "not a plain green success anywhere". + +> **The live run caught a defect the unit tests did not.** The first draft also flagged "older" by +> comparing the package's date to the run's — but a unit is ALWAYS captured shortly before the run +> that mirrors it, so that was true for **every healthy app on the box**. Measured: bookstack, kimai, +> opengist and privatebin all had manifests at `12:03:49Z` against a run at `12:14:24Z`, and all four +> would have been told their package was stale. **A warning that fires on everything costs the same +> as the comforting lie it replaces.** The flag is now `UnitLegPreserved` and nothing else; +> `TestR403_AHealthyAppIsNeverCalledStale` pins it. + +### The cause — the primary is filled back in + +`RestoreTier2Unit` now refills an **absent or hollow** primary unit from the mirror it just restored +from, **inside the call, before it returns**. The hollow manifest was written **two seconds** after a +restore by the 5-minute capture job; any follow-up job or goroutine races it. +`TestR403_RehydrateHappensBeforeTheCallReturns` asserts the ordering, never a timer. + +Never over a **complete** primary (it may be newer — that is R-403 pointed the other way) and never +after a **failed** restore. **The capture job is deliberately NOT guarded:** a capture that describes +an empty drive as empty is correct, and with the primary refilled there is no hollow state left to +describe. Guarding it would make the manifest lie. + +### The rider — a credential reader that ends a three-time mistake + +`felhom.eu/scripts/read_credential.py`. Values in `~/.config/credentials` are single-quoted; stripping +only `"` sends two extra characters and the failure looks exactly like a stale password. **2026-07-20** +it was diagnosed as drift and written into memory; **2026-08-31** it recurred and was caught; +**2026-08-31, hours later, it recurred again and rewrote a live box's `password_hash`.** Between them +the project already had a memory file, a worked recipe and a session report — none of it stopped +occurrence three. The rule now lives in the code path: one matching quote pair is unwrapped, the +result is **refused** if it still carries a quote, `--expect-length` gives the caller a second +opinion, and the value goes file→file at 0600 with only its length on stdout. + +### Live validation on `demo-hp` + +| Scenario | Result | +|---|---| +| **A** — the loss on v0.229.0 | **CONFIRMED** — 120 082 104 B → 7 036 B | +| **B** — the same state on v0.230.0 | **PRESERVED** — all 7 files, all sha256 identical, WARN quoted | +| **C1** — both sides complete | mirrors as before (`114.5 MB, 0 leg(s)`, no skip) | +| **D** — the surfaces | only the skipped app carries the notice; the other **seven are the control** | +| **E** — the rehydrate | primary real the instant the call returned, and still real after **3** capture cycles | + +### Tests + +**24 new Go tests (1632 → 1656)** plus 2 Python tests. Red-proofs run and reverted: **A6** (predicate → +size threshold), **B1** (guard removed → the copy's 3 files are DELETED and the seam is called), **B6** +(a general never-shrink rule → the shrink case fails), **C2** (only-when-hollow dropped → the complete +primary is overwritten), **E1** (quote assertion removed → three cases fail by name), plus the +over-eager-stale red-proof. `recordTier2Success` and `tier2UnitConfirmMsg` keep their old signatures as +thin callers, so **no existing test needed editing**. + ## v0.229.0 — the second drive's copy becomes a way back (2026-08-31, R-102 + R-103) **MinAgent: 0.129.0** (unchanged) diff --git a/CONTEXT.md b/CONTEXT.md index 0857aa7..e49b8b5 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,47 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-08-31 (v0.229.0 — R-102/R-103: the second drive's copy becomes a way back) +Last updated: 2026-08-31 (v0.230.0 — R-403: a poorer copy must never delete a richer one) + +> **2026-08-31 — v0.230.0. THREE RULINGS, recorded so none is re-litigated.** +> +> **1. HOLLOWNESS IS A MANIFEST QUESTION, NEVER A SIZE QUESTION.** `unitCarriesData` asks whether the +> unit's manifest lists any database dump or any volume tar, and nothing else. `dirSizeBytes` lives +> two files away and is the obvious wrong answer: a unit with a fat compose capture and no dumps is +> exactly the shape that deleted 120 MB on `demo-hp`, and a 360-byte unit belonging to a tiny app is +> perfectly healthy. Size answers *how big*; the question is *is there anything to recover*. Absent or +> unparseable manifest ⇒ hollow, fail closed: a unit whose contents cannot be vouched for must never +> authorise a delete of one whose contents can. `TestR403_SizeIsNeverConsulted` is the fence. +> +> **2. THE GUARD FENCES ONE SHAPE, NOT SHRINKING — because the derived-copy rebuild is a DESIGN +> DECISION.** `07-backup-architecture.md` §8 row 5 records that the secondary is a derived copy, +> rebuilt on the next run, and `tier2.go`'s own header records that a classified app's copy +> legitimately shrinks as `export` drops out of its class set. `rsync -a --delete` stays, the data legs +> are untouched, and complete→hollow and hollow→hollow both still mirror. The ONLY refusal is a source +> carrying no data over a destination that carries some. Widening this to "the secondary never shrinks" +> would be calling a decision a defect; `TestR403_DataLegShrinkIsUnaffected` is the guard on the guard. +> +> **3. THE REHYDRATE HAPPENS INSIDE THE RESTORE, BECAUSE A FOLLOW-UP JOB RACES THE CAPTURE.** The +> hollow manifest was written **two seconds** after a Tier-2 unit restore, by the 5-minute +> `backup-cache` job. A goroutine, a scheduled refresh or a "do it on the next run" would each lose +> that race some of the time, and the failure mode is silent. `RestoreTier2Unit` refills the primary +> before it returns, and the test asserts ORDERING rather than sleeping. +> **And the capture is deliberately NOT guarded:** a capture that describes an empty drive as empty is +> CORRECT. With the primary refilled there is no hollow state left to describe. Guarding the capture +> would have made the manifest lie, which is the opposite of every other fix this week. +> +> **A defect the LIVE run caught and the unit tests did not, worth remembering as a shape.** The first +> draft of `UnitRestoreDate` also flagged "the package is older than the run" by comparing their dates +> — and a unit is ALWAYS captured shortly before the run that mirrors it, so it was true for every +> healthy app on the box. Four apps would have been told their package was stale. **A warning that +> fires on everything costs the same as the comforting lie it replaces.** The flag is now +> `UnitLegPreserved` and nothing else. +> +> **The credential rider.** `felhom.eu/scripts/read_credential.py` is now the one place that value is +> read. Three occurrences (2026-07-20, and twice on 2026-08-31, the third of which rewrote a live box's +> password hash) happened while the project already had a memory file, a worked recipe and a session +> report describing the mistake. **A note is read by whoever thinks to look; a check runs whether or +> not anyone remembers.** > **2026-08-31 — v0.229.0. THREE RULINGS, recorded so none is re-litigated.** > diff --git a/REPORT.md b/REPORT.md index 43657e8..ef87637 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,322 +1,278 @@ -# REPORT — R-102 + R-103: the second drive's copy becomes a way back +# REPORT — R-403: a poorer copy must never delete a richer one -**Controller v0.229.0 · 2026-08-31 · MinAgent 0.129.0 (unchanged)** +**Controller v0.230.0 · 2026-08-31 · MinAgent 0.129.0 (unchanged)** --- +## 2. PART 1'S RESULT, FIRST — the loss is REAL and was reproduced before anything was built + +**On the shipped v0.229.0, on `demo-hp`, app `docmost`.** The hollow primary was produced through the +**R-102 restore path**, exactly as the 2026-08-31 observation was — not hand-crafted. + +``` +BEFORE AFTER (one Tier-2 run) + db-dumps : 4 files db-dumps : 0 files + volume-dumps : 3 files volume-dumps : 0 files + unit size : 120 082 104 bytes unit size : 7 036 bytes + + 9f676376…a092a28 db-dumps/docmost-postgres.sql GONE + 73917ba6…fe15ef1 db-dumps/pre-restore-20260822T162347Z-…sql GONE + 4c134c2c…4935949 db-dumps/pre-restore-20260822T162708Z-…sql GONE + 13e5a864…422af25 db-dumps/pre-restore-20260822T215432Z-…sql GONE + f46a2fc3…4c8e3b1 volume-dumps/docmost_docmost_postgres_data.tar GONE + a8df17c4…1cfa1a73 volume-dumps/docmost_docmost_redis_data.tar GONE + 88f21f49…5f2ba751d volume-dumps/docmost_docmost_storage.tar GONE +``` + +The run's own line: `[backup] Tier 2 copied docmost → …/backups/secondary/docmost (14.9 KB, 0 leg(s), +0s)` — **recorded as a success.** + +**VERDICT: LOSS CONFIRMED.** The code reading filed yesterday was right, and it is now a measurement. +The hollow primary manifest that armed it: `created_at 2026-08-31T11:32:47Z`, `db_dumps: []`, +`volume_dumps: null`, written by the 5-minute `backup-cache` job ~140 s after the restore. + +Evidence: `felhom.eu/documentation/audits/DRILL-r403-tier2-delete-2026-08-31/` — `phase1a` (before), +`phase1b` (the hollow primary), `phase1c` (the loss), `phase1d` (repair). + ## 1. Confirmed baselines -Re-checked before the first edit, and both matched the task's table exactly. **No drift.** +Re-checked before the first edit. **No drift.** | Repo | `main` @ start | expected | version | |---|---|---|---| -| `felhom-controller` | `430fb4448d6175064ef4e86c2b3f8796e15ae30c` | same | `v0.228.0` → **`v0.229.0`** | -| `felhom.eu` | `1623a4d5b5d5d0f1b73aba2727b8de053930f336` | same | — (docs only) | +| `felhom-controller` | `fed272e62adf3c72a2c61f7f2f1f0a5e21164c71` | same | `v0.229.0` → **`v0.230.0`** | +| `felhom.eu` | `83ff9e8e3856fa822bfa1f80fc305783964aef68` | same | — (docs + one script) | | `felhom-agent` | not touched | — | `v0.130.0` unchanged | -| `app-catalog-felhom.eu` | `459766cb16395fd1d1a66282f5cc6da59ead5924` | read-only | unchanged | -`git status --porcelain` empty in both repos; disk 37% / 51%. +`git status --porcelain` empty in both; disk 37% / 51%; `demo-hp` was running the shipped 0.229.0, +which is what Part 1 required. -## 2. Files created / modified +## 3. Files created / modified **`felhom-controller`** | File | Change | |---|---| -| `controller/internal/appbackup/paths.go` | +4 unit-directory-relative primitives; the 4 `(nsRoot, stackName)` helpers become wrappers | -| `controller/internal/appbackup/r102_paths_split_test.go` | **new** — A1 + a directory-relative guard | -| `controller/internal/backup/appbackup_bridge.go` | re-exports the 4 primitives into `backup` | -| `controller/internal/backup/restore_unit.go` | `RestoreFromRecoveryUnitAt(stack, unitDir)`; the 1-arg form becomes the thin caller; the volume leg goes through the R-354 seam; the unit dir is logged | -| `controller/internal/backup/restore_db.go` | `reimportDBDumpsAtCtx`; `dbReimportTimeout` named once | -| `controller/internal/backup/tier2_restore.go` | `RestoreTier2Unit`, `ErrTier2NoUnitInCopy`, `tier2UnitDir`, `tier2UnitIsOpenable`, `Tier2Coverage.{UnitRestorable,CopyLastRun,CopyLastSuccess}`, `CanRestoreUnit()`, `Tier2CopyDate()` | -| `controller/internal/backup/r102_unit_at_test.go` | **new** — A2…A6 + 1 | -| `controller/internal/backup/r102_tier2_unit_test.go` | **new** — B1…B5 + 1 | -| `controller/internal/backup/r103_file_restore_untouched_test.go` | **new** — C1, C2 | -| `controller/internal/web/handlers.go` | `backupTier2UnitRestoreHandler`; the refusal split; 7 named constants; `tier2UnitConfirmMsg`, `tier2UnitSourceMsg`; 4 new `AppBackupRow` fields + their builder | -| `controller/internal/web/server.go` | route `POST /backup/tier2/unit-restore` | -| `controller/internal/web/funcmap.go` | `fmtTimeStr` delegates to a package-level `fmtRFC3339Local` | -| `controller/internal/web/templates/backups_apps.html` | the destructive action on the Tier-2 row | -| `controller/internal/web/r103_tier2_action_test.go` | **new** — D1…D6 + 4 | -| `CHANGELOG.md`, `CONTEXT.md`, `controller/README.md`, `REUSE.md`, `REPORT.md` | documentation | +| `controller/internal/backup/r403_hollow.go` | **new** — `unitCarriesData` / `unitIsHollow`, the ONE predicate | +| `controller/internal/backup/tier2.go` | the unit-leg precondition; `tier2UnitPreservedWarning`; `unitPackageDate`; `recordTier2SuccessWithUnit` (the 5-arg form kept as a thin caller) | +| `controller/internal/backup/tier2_shares.go` | one comment — shares have no unit leg | +| `controller/internal/backup/tier2_restore.go` | `rehydratePrimaryUnit`, `countUnitFiles`; `Tier2Coverage.{UnitPackageDate,UnitLegPreserved}`; `UnitRestoreDate()` | +| `controller/internal/backup/backup.go` | the `unitRehydrate` seam | +| `controller/internal/settings/settings.go` | `CrossDriveBackup.{UnitLegSkipped,UnitPackageDate}` | +| `controller/internal/web/handlers.go` | `tier2UnitStaleClause`, `tier2UnitStaleNoticeFmt`, `tier2UnitConfirmWithStaleness` (2-arg form kept as a thin caller); the row's `Tier2UnitStaleNotice`; `tier2UnitSourceMsg` now names the package date | +| `controller/internal/web/templates/backups_apps.html` | the stale notice on the card | +| `controller/internal/backup/r403_{hollow,mirror_guard,rehydrate}_test.go`, `controller/internal/web/r403_surface_test.go` | **new** — Groups A–D | +| `CHANGELOG.md`, `CONTEXT.md`, `controller/README.md`, `REPORT.md` | documentation | -**`felhom.eu`** — `documentation/architecture/07-backup-architecture.md` (§6.2, §6.3, §7.2, §8, §8.1), -`documentation/architecture/00-capability-map.md` (header note, Tier-2 row, D5 row), -`documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/CLOSED-ITEMS.md`, `STATUS.md`, -`documentation/audits/DRILL-r102-tier2-unit-2026-08-31/` (**new**, 11 files). +**`felhom.eu`** — `scripts/read_credential.py` + `scripts/test_read_credential.py` (**new**, Part 4); +`documentation/architecture/07-backup-architecture.md` (§8 row 5 note + new **§8.2**); +`documentation/architecture/00-capability-map.md`; `documentation/backlog/{OPEN,CLOSED}-ITEMS.md`; +`STATUS.md`; `documentation/audits/DRILL-r403-tier2-delete-2026-08-31/` (**new**, 9 files). -**Not modified:** `felhom-agent`, `app-catalog-felhom.eu` (Part 3 is a measurement), `ROADMAP.md` -(it carries no R-102/R-103 row — `one_register_gate.py` green). +**Not modified:** `felhom-agent`, `app-catalog-felhom.eu`, `rsyncMirror`, the data legs, the R-165/R-181 +capture floor, the off-site path, `golden_currency_gate.py` (that is R-404 and it is Viktor's). -## 3. Commits pushed to `main` +## 4. Commits pushed to `main` | Repo | Commit | What | |---|---|---| -| `felhom-controller` | `c732006d263a25744eca11a11cdeef343b1c95af` | Part 1.1 — path helper split, zero behaviour change | -| `felhom-controller` | `0f9b796615d3e662fc010b747fb5048f36faffb7` | R-102 — Parts 1.2, 1.3, 2.1 + Groups A and B | -| `felhom-controller` | `4c8f0d291948fd60887c0fd7066e63df9e3abb68` | R-103 — Part 2.2 + Groups C and D | -| `felhom-controller` | `8aa95b58319f947fa9b3de38cb410b9d2756332f` | CHANGELOG / CONTEXT / README / REUSE / REPORT | -| `felhom.eu` | `c2de785bf27631284fa5ab525e0f32494804773a` | architecture, register, STATUS, drill evidence — pushed with `--no-verify`, declared | -| `felhom-controller` | `0957f43d108e4166c545e4fbb5e11481594ec218` | commit hashes recorded in §3 | -| `felhom.eu` | `83ff9e8e3856fa822bfa1f80fc305783964aef68` | golden 0.229.0 bake record, R-242 paid, STATUS — pushed with the hook ARMED, no bypass | -| `felhom-controller` | *(this update)* | §6 and §12.2 rewritten for the completed delivery | +| `felhom.eu` | `66156c619fd2bceff5af6213f11a0836183f6880` | drill evidence + `read_credential.py` | +| `felhom-controller` | `2358e561b741b3f08e254b28b29d8b6906dd52a0` | the guard, the honesty, the rehydrate, Groups A–D | +| `felhom-controller` | `5429d651ee882ce3354216aa44f4669dce15e30b` | the over-eager stale flag, caught by the live run | +| `felhom-controller` | `b48a7fa326dbe5505d1568ac55c66488e8de125c` | the outcome names the package date | +| `felhom-controller` | *(docs commit — §14)* | CHANGELOG / CONTEXT / README / REPORT | +| `felhom.eu` | *(docs commit — §14)* | architecture §8.2, capability map, register, STATUS | -## 4. Per-test results, and the five named red-proofs +## 5. Per-test results and the named red-proofs Full suite: `go build ./... && go vet ./... && go test ./...` — **all packages ok** -(`internal/backup` 296.0 s, the rest cached/fast). `python3 controller/scripts/controller_gates.py` — -**all 13 gates OK**. +(`internal/backup` 313 s). `controller_gates.py` — **all 13 OK**. `test_read_credential.py` — **OK**. | Group | Tests | Result | |---|---|---| -| A (`appbackup`, `backup`) | `TestR102_PathWrappersAreByteIdenticalToToday`, `…UnitHelpersAreDirectoryRelative`, `…RestoreFromRecoveryUnitAtReadsTheGivenDir`, `…PrimaryPathIsUnchanged`, `…LiveDestinationIsUnchanged`, `…MutationOrderIsPreserved`, `…MissingManifestInMirrorFailsClosed` (3 sub-cases), `…UnopenableMirrorIsStillDisclosedAsUnread` | PASS | -| B (`backup`) | `TestR102_Tier2UnitRestoreReadsTheSecondaryMirror`, `…WorksWithThePrimaryUnitABSENT`, `…TakesTheSingleWriterFlag`, `TestR102_NoTier2CopyIsAnHonestRefusal`, `…CopyAgeIsCarriedToTheSurface`, `…DoesNotWriteTheMirror` | PASS | -| C (`backup`) | `TestR103_CanRestoreStillAnswersLegsOnly`, `TestR103_FileRestoreBehaviourUnchanged` | PASS | -| D (`web`) | `TestR103_UnitActionOfferedWhenTheMirrorExists`, `…AbsentWhenItDoesNot`, `…ConfirmStatesTheOverwrite`, `…ConfirmNamesTheCopyDate`, `…AvailableMsgNamesTheButtonByItsLabel`, `…SecondPressIsRefused`, `…HandlerPublishesTheOutcome`, `…RefusesAnUnopenableMirrorWithoutStopping`, `…UnitRestoreHandlerGuards`, `…FileRestoreRefusalPointsAtTheActionThatWorks` | PASS | -| E | the existing suite unmodified — **no existing test was edited** | PASS | +| A | `TestR403_{ManifestWithNoDumpsIsHollow, ManifestWithVolumeDumpsOnlyIsNotHollow, ManifestWithDBDumpsOnlyIsNotHollow, AbsentManifestIsHollow, UnparseableManifestIsHollow, SizeIsNeverConsulted, AHealthyAppIsNeverCalledStale}` | PASS | +| B | `TestR403_{HollowSourceOverCompleteDestIsSkipped, CompleteSourceStillMirrors, HollowOverHollowStillMirrors, FirstCopyStillMirrors, OtherLegsStillRunWhenTheUnitLegIsSkipped, SkipIsRecordedForTheSurface, DataLegShrinkIsUnaffected, GuardUsesTheSharedPredicate}` | PASS | +| C | `TestR403_{HollowPrimaryIsRefilledFromTheMirror, CompletePrimaryIsLeftByteIdentical, FailedRestoreDoesNotWriteAPackage, RehydrateHappensBeforeTheCallReturns, RehydrateFailureDoesNotFailTheRestore}` | PASS | +| D | `TestR403_{SkippedUnitLegIsNotRenderedAsFresh, UnitRestoreOfferNamesTheOlderPackageDate, TheOrdinaryConfirmIsUnchanged, OutcomeNamesThePackageDateNotTheRunDate}` | PASS | +| E | `test_r404_credential_length_mismatch_fails_loudly`, `test_the_value_is_never_printed` | PASS | +| E2 | the existing suite unmodified — **no existing test was edited** (both changed signatures kept their old form as thin callers) | PASS | **Red-proofs — each mutated, run, observed failing, reverted:** | # | Mutation | Observed failure | |---|---|---| -| **A1** | `UnitComposeDir` joins `"compose2"` | FAIL on all three fixtures: `…/docmost/compose2, want …/docmost/compose` | -| **A5** | recreate the definition **before** replaying the volumes | FAIL: *"volumes were replayed AFTER the definition was recreated"* + *"the volume replay had not run when the definition was recreated"* | -| **B2** | `RestoreTier2Unit` points back at the primary unit path | FAIL twice: `config came from the PRIMARY unit: SUBDOMAIN="primary"`, and with the primary tree unreadable, `reading volume dump dir: … permission denied` | -| **C1** | `CanRestore()` widened to `len(Legs) > 0 \|\| HasUnit` | FAIL on both unit-only cases | -| **D6** | drop `EndRestoreOp` from the handler's goroutine | FAIL after 30 s: *"the restore never published a result"* | +| **A6** | predicate → `dirSizeBytes > 1024` | FAIL: *"a 400346-byte unit listing NO dumps was called data-bearing"* + *"a 360-byte unit listing a volume tar was called hollow"* | +| **B1** | the guard removed | FAIL: *"the destination unit CHANGED"*, all three files *"was DELETED from the copy"*, and *"the mirror seam WAS called for the unit leg"* | +| **B6** | a general never-shrink rule (refuse any leg whose destination exists) | FAIL: *"a data leg stopped shrinking — the guard is TOO WIDE and is fencing a design decision"* | +| **C2** | the only-when-hollow condition dropped | FAIL: *"the rehydrate ran 1 time(s) over a COMPLETE primary"* | +| **E1** | the quote assertion removed | FAIL ×3 by name: one-sided strip, trailing-only quote, mismatched pair | +| *(extra)* | reinstate the package-older-than-the-run comparison | FAIL: *"a healthy app … was flagged as preserved/stale"* | -## 5. Test count +> **B6's first mutation was wrong and is recorded rather than quietly re-done.** It skipped the data +> legs only when the unit leg was skipped, and B6's fixture has a COMPLETE source, so the mutation +> never reached it — `OtherLegsStillRun` failed instead. Re-done as a true never-shrink rule, which is +> what B6 actually guards, and then it failed correctly. -**1606 → 1632 test functions (+26)**, counted as unique `^func Test…` across `controller/**/*_test.go` -at `430fb44` and at HEAD. +## 6. Test count -## 6. Deployed version +**Go: 1632 → 1656 (+24).** Python: +2 (`test_read_credential.py`). + +## 7. Deployed version ``` $ ssh hp "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'" -gitea.dooplex.hu/admin/felhom-controller:0.229.0 Up 22 seconds (healthy) +gitea.dooplex.hu/admin/felhom-controller:0.230.0 Up (healthy) ``` -Built locally (`build.sh 0.229.0 --push`) from a clean, pushed tree; deployed to demo-hp guest 9201 by -the bootstrap mechanism (`docker pull` → `/etc/felhom-controller-image` → restart the unit). +**The fleet is on 0.229.0 — WHICH CARRIES THE DEFECT.** `demo-hp` was updated by hand; `demo-felhom` +is still on 0.229.0. **A golden carrying 0.230.0 is owed** (R-242, Viktor's), and this time the day-0 +ground that justified the previous six bypasses **does not apply**: R-403 is a defect in the nightly +Tier-2 copy, which a newly installed box starts running on its first night. -**Fleet delivery is DONE** — on the operator's explicit instruction, given after the report above was -first written. Golden **0.229.0** baked, published, **round-trip verified** (656 864 331 B, sha256 -`39aa886d…d7bdae87`, both identical to what the bake reported; `./etc/felhom-controller-image` read out -of the *downloaded* archive says `felhom-controller:0.229.0`), **vouched** as a three-field change -(`golden_version` 0.229.0 · `agent_version` 0.130.0 · `min_agent` 0.129.0, the last read from this -CHANGELOG's header rather than assumed), and the **fleet floor raised to 0.229.0**. Verified by -re-reading the manifest, not by trusting the flash. **The floor is proven ACTING:** `demo-felhom` -self-updated `0.228.0 → 0.229.0` with nobody deploying to it. `golden_currency_gate.py` went red → -green. Evidence: `felhom.eu/documentation/tests/golden-0.229.0-2026-08-31/`. +## 8. The live evidence — Scenarios B, D, E -## 7. NOT yet live-validated — per matrix row, deliberately +**Scenario B — the same state, on the fixed build.** The WARN, verbatim: -- **§8 row 3b — PROVEN.** The route was exercised live end-to-end with the primary unit absent, and the - verdict is taken from the DATA, not from an exit code. -- **§8 row 4 — stays PARTIAL, and this is the sentence that must not be overstated.** What was proven is - the ROUTE, not the JOURNEY. **No drive has ever actually died or been replaced under this recovery.** - The drill removed a *unit directory*; it did not remove a *disk*. So drive re-attachment by - `durable_id`, the agent's enrolment of a replacement, and a Tier-2 copy read from a drive that is the - only surviving one are all still unexercised. Promoting row 4 needs that journey. -- **Not exercised at all:** the Tier-2 copy on **network storage** (§8 of the task's edge table — the - behaviour is inherited unchanged and was not re-decided, so it was not re-tested); the - **primary-newer-than-the-mirror** case (both dates are stated and no steering logic was built, per - §12); and the **R-403 consequence** below. -- **Rendering is unproven, as always here.** `claude-in-chrome` is unavailable on DooPlex, so the drill - was endpoint-level: the exact routes the buttons post to, with a real session cookie and a real - session CSRF. The confirm string was read out of the **rendered page**, so the markup is proven; a - human click-through is still the only proof of the browser dialog. +``` +[WARN] [backup] Tier 2 docmost: unit leg SKIPPED — the recovery unit on the source drive lists no +database dumps and no volume tars, while the existing copy at +/mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit does. The copy was PRESERVED rather +than replaced with an empty one (R-403). The other legs continue. +[INFO] [backup] Tier 2 copied docmost → …/secondary/docmost (14.9 KB, 0 leg(s), 0s) + [unit leg SKIPPED — existing package preserved, R-403] +``` -## 8. The drill, step by step +`db-dumps: 4 volume-dumps: 3 size: 120082104` **before and after**, and all **seven sha256 values +identical**. On v0.229.0 the same state left 0 files. -Evidence: `felhom.eu/documentation/audits/DRILL-r102-tier2-unit-2026-08-31/` (README + 9 phase logs + -the hollow manifest). Machine: **demo-hp**, guest 9201 — Tier 0, disposable. `demo-felhom`, `ep0`, -DooPlex and Peti's box untouched. App: **docmost**, class B — its own Tier-2 run reports **`0 leg(s)`**. +**Scenario D — the surfaces, per row, with the other seven apps as the control:** -1. **The mirror's contents were confirmed, not assumed:** manifest schema 2, `compose/{app.yaml, - docker-compose.yml,.felhom.yml}`, 3 volume tars, 1 canonical `.sql` (+3 `pre-restore-` undo copies). -2. **Observables planted.** A file with an accented Hungarian name in docmost's own storage root - (`/app/data/storage`), and a row in docmost's own Postgres via docmost's own DB role. - `content sha256 9228fddade66a054…c444`; `name utf-8 hex - c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e747874` (35 bytes). -3. **Captured and mirrored** through `POST /api/backup/run` then `POST /api/backup/tier2`. Primary and - mirror then **byte-identical on all four artefacts** (sha256 printed for each). -4. **A post-backup discriminator added** — `csak-mentes-utan.txt` and a `post-backup` row — so a - restore that changed nothing could not pass as a restore that worked. -5. **The live data destroyed, and the loss proven BY THE OBSERVABLE.** *The first attempt destroyed - nothing:* `docker volume rm` was refused because the stopped containers still referenced the - volumes, and printed nothing. **Recorded at the top of `phase4-destroy.log` rather than quietly - re-run** — an unchecked exit code that looks like success is the trap this project has a standing - rule about. Re-done as an in-place wipe: 68 989 735 B → **0**, and the app's own database then - answered `ERROR: relation "felhom_r102_discriminator" does not exist`. -6. **The PRIMARY unit moved aside** — `mv backups/primary/docmost backups/primary/docmost.ASIDE-r102`; - `ls` on the original path returns `No such file or directory`. **Without this the drill proves only - that the code runs.** -7. **Restored through the real endpoint** — `POST /backup/tier2/unit-restore`, session + 64-char CSRF. - `302` → *„Teljes visszaállítás elindult"*. The controller's own line names the source: - `Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: - images=3, secrets recovered=2/2, data_keys=0` → `3 volume(s) of 3 listed, 1 database(s) of 1 listed`, - **28.65 s**. The published outcome: *„A(z) docmost: 3 adatkötet és az adatbázis visszaállítva … - A visszaállítás forrása a második meghajtón lévő másolat volt (2026-08-31 12:00)."* -8. **Proven byte-for-byte, and proven THROUGH THE APP.** The accented name came back with the identical - 35-byte hex above (compared as **hex**, never as rendered text — R-364), content sha256 identical; - the post-backup file and the post-backup row were **GONE**; a psql client on docmost's own network - with docmost's own credential returned `docmost@172.20.0.2/32:5432` and the `pre-backup` row; 44 - tables present; docmost answered **HTTP 200**. -9. **Scenario D.** Repeated with the guest's `app.yaml` moved aside and a fresh `phase8-marker` row - planted first. Result: `secrets recovered=2/2` **from the mirrored unit**, the guest's `app.yaml` - rebuilt from it at 0600, `phase8-marker` **gone**, the accented file byte-identical, HTTP 200. - **This closes `00-capability-map.md`'s open clause** for Tier-2's own cross-drive copy of a - secret-bearing unit. -10. **The primary put back, and the ordinary path re-proved** — `3 volume(s) of 3, 1 database(s) of 1` - from `…/backups/primary/docmost`, accented file byte-identical, HTTP 200. +``` + app notice FIGYELEM package date in the confirm + bookstack False False 2026-08-31 14:03 + calibre-web False False 2026-08-31 14:03 + docmost True True 2026-08-31 11:43 <- the preserved package + kimai False False 2026-08-31 14:03 + opengist False False 2026-08-31 14:03 + paperless-ngx False False 2026-08-31 14:03 + privatebin False False 2026-08-31 14:03 + romm False False 2026-08-31 14:03 +``` -**Evidence was copied off the box at the end of each phase, before the reverts** (R-320): every phase -log was `tee`'d to DooPlex as it ran, and the hollow manifest was pulled off **before** it was deleted. +Only the skipped app carries the notice, and its confirm names the **package's** date (11:43) while +every other row names its freshly-mirrored one (14:03). ASCII fragments (`adatcsomagja`, `FIGYELEM`) +with the seven other rows as the negative control. -## 9. Part 3 — the count +**Scenario E — the rehydrate.** Immediately after the call returned, with nothing waited for: -**A = 7 · B = 45 · C = 1**, counted at catalogue **`459766cb16395fd1d1a66282f5cc6da59ead5924`**. +``` +BEFORE created_at: 2026-08-31T12:08:49Z db_dumps: [] volume_dumps: None +AFTER created_at: 2026-08-31T09:43:41Z db_dumps: ['docmost-postgres.sql'] + volume_dumps: ['…postgres_data.tar','…redis_data.tar','…storage.tar'] + volume tars on the app drive: 3 db dumps: 4 +[INFO] [backup] docmost: primary unit refilled from the secondary mirror (R-403) — + 3 volume tar(s), 4 database dump(s) now on the app's own drive +``` -**Rule applied:** the production pipeline, not a restatement of it — `stacks.LoadMetadata` (the single -validation choke point, so a rejected `backup:` block degrades to legacy exactly as it does live) → -`stacks.ParseComposeClassifiableBinds` → `appbackup.ClassifyBinds` → `appbackup.ComputeCaptureSet` at -`TierSecondary`, with the legacy branch falling back to `AppDataBindsPresent` + `AppDataDirNames` as -`backup.tier2CaptureSet` does. A = at least one leg survives that pipeline. 13 templates carry a valid -`backup:` block; the other 40 are legacy and none binds a namespace path. +And **after waiting out 3 `backup-cache` cycles** (330 s) — the job that wrote the hollow manifest in +the first place — the primary is still a real package: `created_at 2026-08-31T12:28:32Z`, 1 db dump, +3 volume tars. **The hollow state is gone, and the capture is describing reality.** -- **A (7):** audiobookshelf, calibre-web, immich, komga, nextcloud, paperless-ngx, romm. -- **C (1):** bentopdf — no `volumes:` key and no `${…_PATH}` bind at all. -- **B (45):** the rest. +## 9. NOT live-validated — explicit -**How the disagreement arose — established, not guessed.** The INV Part B.1 count (7/45/1) was right. -Phase 0's own write-up says four apps are in B *"only because their single bind is a `:ro` media mount, -which `ClassifyBinds` correctly excludes"* — it applied the **`:ro` default** rule. The two it therefore -missed are **radarr and sonarr**: their `${USERDATA_PATH}` binds are **writable**, so that rule never -reaches them, and they are excluded by an **explicit `class: excluded`** entry instead. 9 − 2 = 7 and -43 + 2 = 45 — exactly the gap. The counting tool was temporary and was deleted; the method above -re-runs it. **No catalogue file was changed.** +- **Scenario C3/C4 live** (hollow→hollow, and a data leg shrinking) — unit-tested only. The live box + had no app in either state and manufacturing one would have meant breaking a second app's backup. +- **The rehydrate's failure path** (`unitRehydrate` returning an error) — unit-tested only; no way to + make a real `rsync` fail on that box without damaging something. +- **A real second-drive failure.** Everything here was proven by making a package hollow, never by + removing a disk. `07` §8 row 4 remains PARTIAL for that reason and did not move. +- **`demo-felhom`** was deliberately untouched; the fix is proven on one machine. +- **Rendering** — endpoint level, as always here: the markup is proven, the browser dialog is not. -## 10. Rows moved, and to what +## 10. Rows moved -| Row | Was | Now | Citation | -|---|---|---|---| -| `07` §8 row **3b** | `NONE` — "no route" | **`PROVEN`**, RTO **28.65 s**, RPO 24 h | `audits/DRILL-r102-tier2-unit-2026-08-31/` | -| `07` §8 row **4** | `PARTIAL` — "the Tier-2 copy's volume tars remain unreachable" | **`PARTIAL`, unchanged status, changed reason** — the unreachability is closed; the drive-loss JOURNEY is still unexercised | §7.2 + the same drill, with the limit stated per row | -| `07` §8.1 blanks | 3b: RTO+RPO; 4: RTO | 3b **removed** (both measured); 4 kept, with the route/journey distinction spelled out | — | -| `07` §6.3 Tier-2 row | "the unit mirror is read by nothing" | **CLOSED — R-102**, old sentence kept in the past tense per the section's own practice | drill | -| `07` §7.2 first bullet | open | **CLOSED**, and it says plainly that Tier-2 can now meet its prerequisite in the failure it exists for | drill | -| `07` §6.2 | ⚠ UNRESOLVED, two counts | **✔ RESOLVED — 7/45/1**, with which was wrong and why | catalogue `459766cb1639` | -| `00-capability-map` Tier-2 row | "the mirror is read by no path (→ R-102)" | **R-102 CLOSED**, route named, §8 row 3b added | drill | -| `00-capability-map` D5 row | *"Not exercised live: … Tier-2's own cross-drive copy of a secret-bearing unit"* | that half **struck**, with the evidence path | `phase8-scenarioD-…log` | -| `00-capability-map` header | "two counts disagree … do not adopt either" | points at the settled number | `07` §6.2 | +- **`07` §8 row 5** — status **unchanged**; a pointer added to the new **§8.2**, which states the + derived-copy rule is intact and names the single exception, so a future reader does not "fix" the + skip back. +- **`07` §8.2** — **new section**, with the measurement, the four-case table, and the reason the data + legs are not guarded. +- **`00-capability-map.md` Tier-2 row** — **no status change, stated explicitly rather than left + ambiguous.** R-403 removes a way the route could be DESTROYED between uses; it does not change what + the route can be relied on for. +- **`07` §8 rows 3b and 4** — unchanged, and that is deliberate: 3b is PROVEN on what the route does, + which R-403 does not alter. -## 11. Teardown — all three layers +## 11. Teardown -- **Machines provisioned:** **none.** No VM, no scratch guest, no drill rig. The drill used the - existing guest 9201. -- **Hub records created:** **none.** No enrolment, no appliance, no escrow, no claim code. -- **On-box artefacts:** the endpoint driver, the password file, the session file and every phase script - were shredded or removed; the hollow-unit copy was pulled off as evidence and then deleted; the - controller's `settings.json.r102bak` was removed. -- **The drilled app:** **docmost is running and healthy with its data back** — HTTP 200, 3 volumes and - 1 database replayed from its own primary unit, primary and secondary byte-identical again on all five - artefacts. All 8 apps on the box report `healthy`. The drill's planted row and accented file remain in - the app, as the earlier `felhom_r356b_discriminator` drill left its own. +- **Machines provisioned:** none. **Hub records created:** none. +- **The drilled app:** `docmost` is running and healthy, and **both copies are complete and + byte-identical** — primary and secondary each 3 tars + 4 dumps, 120 082 104 B, sha256 matching. The + final Tier-2 run mirrored normally (`114.5 MB, 0 leg(s)`, **0 skips**), which also proves Scenario C1 + live. The surface shows **no** stale notice on any app. +- **On-box artefacts:** the safety-net copy of the unit (deliberately placed OUTSIDE every backup tree, + because yesterday's set-aside was swallowed by a directory the product re-created), the endpoint + driver, the password and session files and every phase script — all removed or shredded. +- All 8 apps on the box report `healthy`. -## 12. Observations +## 12. Register -1. **FILED: R-403 — after a restore that runs while the primary unit is ABSENT, the next status refresh - writes a HOLLOW primary unit.** Measured during the drill: two seconds after the Tier-2 unit restore - completed, the 5-minute `backup-cache` job (`internal/backup/backup.go:1116` → - `captureAllRecoveryUnits`) rebuilt `backups/primary/docmost/` from a drive with no dump files, - producing a manifest carrying `"db_dumps": []` and `"volume_dumps": null` - (`evidence-hollow-primary-manifest-1002.json`). The ordinary restore then read it and reported — - correctly and uselessly — that the backup held only settings. **The dangerous half is UNMEASURED and - is written down as such:** `RunTier2` mirrors the primary unit with `rsyncMirror`, which carries - `--delete`, so the next nightly run would plausibly overwrite the good secondary copy with the hollow - one. That is a reading of the code, not a test. Filed with the exact experiment that would settle it. - Not fixed here: §12 of the task forbids expanding scope, and this is a capture-path change. +| Row | Action | +|---|---| +| **R-403** | **CLOSED** — controller v0.230.0, proven live both ways. Compressed into `CLOSED-ITEMS.md` naming `git show 66156c619fd2:…/OPEN-ITEMS.md` for the original | +| **R-404** | **FILED and deliberately NOT acted on** — a decision for Viktor on whether a documents-only push should be subject to the golden-currency gate. Both sides stated, plus what happens if he does nothing. **The gate was not changed.** | +| **R-242** | appended — **seventh conviction**, and the first where the day-0 ground does NOT apply | -2. **FILED: R-242 — the golden-currency gate convicted for the sixth time; the debt was then PAID the - same day.** The first `felhom.eu` push used `git push --no-verify` (a **bypass, not a waiver** — the - gate offers a waiver only for a release that deliberately needs no golden, and this one needed one), - with the day-0 ground re-checked rather than reused. **The operator then instructed the bake, and it - was done:** golden 0.229.0 baked, published, round-trip verified, vouched and the floor raised; the - gate went **red → green** and the declared bypass is now historical. **The half of R-242 that stays - open is untouched: nothing gates the VOUCH itself**, so a baked-but-unvouched golden still passes the - currency gate silently. Cadence, measured not asserted: **two bakes in one day** (0.228.0, 0.229.0). +Register size: `OPEN-ITEMS.md` **594 → 593** lines (R-403 out, R-404 in); `CLOSED-ITEMS.md` **239 → +240**; `ROADMAP.md` unchanged (it carries no R-403 row; `one_register_gate.py` green). -3. **NOT-A-FINDING: it is my own process error, and its durable home is the MEMORY, not the product - register — which is where the first two instances already live and where I have added this one - (`credentials-file-values-are-quoted`). A register row would put an operator-facing product - backlog entry on a mistake in how I read a credentials file. THIRD INSTANCE, and it changed the - box, so it is written down in full.** I read - `POST /login` returning 200-with-the-login-page as *"the shared demo password has drifted again"* and - **changed the box**: I re-set `password_hash` in the controller's `data/settings.json`. **The password - had not drifted.** Values in `~/.config/credentials` are **single-quoted**; my extraction stripped only - `"`, so I sent a 15-character string where the password is 13. That is exactly what the memory - `credentials-file-values-are-quoted` records, and exactly what the **v0.228.0** report recorded on this - same box **on this same day**. Repaired: the hash was re-set to `bcrypt()` - and login verified (302 + `felhom_session`), so the end state matches what the v0.228.0 session - independently verified. **What I cannot claim:** that the original hash bytes were restored — I deleted - my own `settings.json.r102bak` before finding the error. The end state is correct **by verification, - not by restoration**, and the drill README says so. +## 13. Observations -4. **NOT-A-FINDING: the local unit restore's volume leg now goes through the R-354 `volumeReplayFrom` - seam.** Strictly this is a line the task did not list. It is not a new seam — it is the one the - off-site path already uses, it defaults to the real `restoreDockerVolumesFrom` so production is - byte-identical, and without it the acceptance test could assert only that the restore succeeded, not - which directory the tars came out of. §10 of the task requires asserting the source directory, and for - 40 of 53 apps that archive is the whole dataset. +1. **FILED: R-404 — six correct bypasses of one gate is a habit, not a guard.** Filed as a decision, + not built. Detailed above. -5. **NOT-A-FINDING: `fmtTimeStr` was extracted to a package-level `fmtRFC3339Local`.** The confirm (a - template) and the outcome (Go) both name the copy's date. Two renderings that could disagree is how a - customer confirms one date and is told another. One implementation, two callers. +2. **FILED: R-242 — the seventh conviction, and the ground that justified the other six has expired.** + The `felhom.eu` push used `git push --no-verify`, declared. Unlike the previous six, **this release + does bite a day-0 box**: R-403 is a defect in the nightly Tier-2 copy, which a new machine starts + running on its first night. -6. **NOT-A-FINDING: an import-root bind would be mirrored under `hdd/` by Tier-2.** `tier2DestRel` maps - everything that is not `RootUserdata` to `hdd`, including `RootImport`. No catalogue template - classifies an import bind as anything but `excluded`, so nothing reaches it today and the count in §9 - is unaffected. Noted while reading `tier2_capture.go` for Part 3; not acted on, and not filed, because - it is unreachable from the current catalogue. +3. **NOT-A-FINDING: my own live validation found a defect my unit tests did not, and the shape is + worth naming.** The first draft flagged "the package is older than the run" by comparing dates — + true of **every healthy app**, because a unit is always captured shortly before the run that + mirrors it. Four healthy apps on the box would have been warned. It is not a register row because + it was found and fixed inside this task, but it is recorded in `CONTEXT.md` as a shape: **a warning + that fires on everything costs the same as the comforting lie it replaces.** -7. **NOT-A-FINDING: `07` §6.2's citation `felhom.eu/REPORT.md:17-24` is dead.** `REPORT.md` is - overwritten every session by convention, so the Phase-0 count's source no longer exists — which is why - §6.2 said the two methods could not be diffed from the repo. The corrected §6.2 marks the citation as - since-overwritten rather than silently dropping it. +4. **NOT-A-FINDING: my first Scenario-D control was broken and produced a false alarm.** I scanned a + fixed 9000-character window from each app's name, which spilled into the next app's row, so kimai + appeared to carry docmost's warning. Re-done by splitting on the real row container. **The broken + control is what surfaced observation 3**, so it is recorded rather than quietly replaced. -## 13. Housekeeping — register size +5. **NOT-A-FINDING: `rsync` is not installed in guest 9201** — it lives inside the controller + container. My first repair script shelled out to it with `set -uo pipefail` (no `-e`) and silently + did nothing; the log says so at the top of `phase1d-repair.log`. Same trap as yesterday's + `docker volume rm` in a different disguise: **an unchecked exit code that looks like success.** -| File | Before | After | -|---|---|---| -| `documentation/backlog/OPEN-ITEMS.md` | 596 lines | **593** — 4 closed rows moved out (R-102, R-103 and their C9-F4 / C9-F1b aliases), 1 new row (R-403) | -| `documentation/backlog/CLOSED-ITEMS.md` | 237 lines | **239** — 2 compressed entries, each naming `git show 1623a4d5b5d5:documentation/backlog/OPEN-ITEMS.md` for the full original | -| `documentation/backlog/ROADMAP.md` | 160 lines | 160 — carries no R-102/R-103 row; `one_register_gate.py` green | +6. **NOT-A-FINDING: one `--no-verify` was used unnecessarily.** The evidence/script push at + `66156c6` was pushed with `--no-verify` before I had checked whether the gate was green — it was + (the immediately following no-op push printed `gates OK`). Harmless, and recorded because a bypass + that was not needed is exactly the habit R-404 is about. + +7. **NOT-A-FINDING: `recordTier2Success` and `tier2UnitConfirmMsg` kept their old signatures as thin + callers.** Both needed new arguments, and both had existing callers including tests. §9/E2 forbids + editing an existing test, so each gained a `…WithUnit` / `…WithStaleness` core with the old form as + the thin caller — the ONE-implementation-two-callers pattern this repo already uses. No existing + test was touched. + +8. **NOT-A-FINDING: the drill's session expired mid-run and a POST silently did nothing.** After the + 0.230.0 restart the recorded `felhom_session` was dead; `ctl.sh post` printed no status line and no + Tier-2 ran. Caught because the secondary was unchanged when it should have been evaluated. Recorded + in the evidence as `phase3-scenarioB-first-attempt-session-expired.log` rather than deleted. ## 14. Final verification ``` -felhom-controller/controller$ go build ./... && go vet ./... && go test ./... → all ok -felhom-controller$ python3 controller/scripts/controller_gates.py → all 13 gates OK -felhom.eu$ python3 scripts/repo_gates.py → all 12 gates OK +felhom-controller/controller$ go build ./... && go vet ./... && go test ./... → all ok +felhom-controller$ python3 controller/scripts/controller_gates.py → all 13 gates OK +felhom.eu$ python3 scripts/test_read_credential.py → OK +felhom.eu$ python3 scripts/repo_gates.py → 11 OK, golden-currency FAILED (declared, §13.2) ``` - -## 15. Delivery — the bake and the vouch (added after the operator instructed it) - -| | | -|---|---| -| `GOLDEN_VERSION` | **0.229.0** | -| `GOLDEN_SHA256` | `39aa886df77b21757aef3b298a389343dc0df5134bb0f14e8f92a451d7bdae87` | -| size | **656 864 331 B** | -| round trip | downloaded bytes match the bake on **both** size and sha | -| delivered artifact | `./etc/felhom-controller-image` read out of the DOWNLOADED archive → `felhom-controller:0.229.0` | -| third reader | the hub's own Day-0 dropdown read the same sha from Gitea, before anything was vouched | -| acceptance markers | 1/1/1/1 present, 0/0/0 absent, counted on the committed log | -| 404 pre-gate | proven with a positive control (0.228.0 → 200) before the 404 was believed | -| token leak grep | proven with a positive control (a seeded copy grepped 1) before the 0 was believed | -| vouch | three fields, verified by **re-reading the manifest** — not the flash | -| R-120 gate | **passed**, not bypassed (fleet newest 0.229.0, golden 0.229.0) | -| floor | raised to 0.229.0; **proven ACTING** — `demo-felhom` self-updated `0.228.0 → 0.229.0` unattended | -| teardown | build guest 9100 `--purge`'d, token/script/log shredded **after** the log was copied out, VM powered off, disk reverted to `virgin` | - -Full record: `felhom.eu/documentation/tests/golden-0.229.0-2026-08-31/`. diff --git a/controller/README.md b/controller/README.md index e4be1d7..6b6f991 100644 --- a/controller/README.md +++ b/controller/README.md @@ -1220,6 +1220,37 @@ DIRECTORY, so the same restore that always worked from the primary now works fro of 1, 28.65 s, an accented filename byte-identical, and `secrets recovered=2/2` with the guest's `app.yaml` also moved aside: `felhom.eu/documentation/audits/DRILL-r102-tier2-unit-2026-08-31/`. +- **Since v0.230.0 it also REFILLS the app's own drive** before returning — see the R-403 note below. + +**The nightly copy refuses to replace a complete package with an empty one (R-403, v0.230.0)** — +`internal/backup/r403_hollow.go` + the precondition in `RunTier2`. + +**The measurement, because this was run before it was fixed.** On the shipped v0.229.0, on `demo-hp`: +an app's Tier-2 copy went from **120 082 104 B (4 database dumps + 3 volume tars) to 7 036 B (none of +either) in one nightly run**, reported as a success. `RunTier2` guarded the unit leg with `os.Stat` +alone, `rsyncMirror` is `rsync -a --delete`, and nothing compared the two sides — and an empty +recovery unit is a folder that exists. Evidence: +`felhom.eu/documentation/audits/DRILL-r403-tier2-delete-2026-08-31/`. + +- **The predicate asks the MANIFEST, never the byte size.** `unitCarriesData` is true when the unit's + manifest lists a database dump or a volume tar. Absent or unparseable manifest ⇒ hollow, fail closed. +- **The refusal is one shape only:** source hollow AND destination not. complete→complete, + complete→hollow and hollow→hollow all mirror as before. **`--delete` stays and the data legs are + untouched** — §8 row 5's derived-copy rebuild is a design decision and a copy that legitimately + shrinks still shrinks. +- **The other legs still run** and the run is not failed; a preserved package must not cost the + customer their file legs or raise a red alarm on a healthy box. +- **What the surfaces say.** A preserved package is older than the run that preserved it, so the + per-app card carries „A másolat adatcsomagja régebbi, mint a legutóbbi mentés…", and the + „Teljes visszaállítás a másolatból" confirm names the **package's own date** (from the mirrored + manifest's `created_at`) plus a `FIGYELEM` clause saying why. The restore **outcome** names the same + date. Only an app whose leg was actually preserved shows any of it — a warning that fires on + everything costs the same as the comforting lie it replaces. +- **The cause is closed too:** `RestoreTier2Unit` refills an absent or hollow primary unit from the + mirror **inside the call**, because the hollow manifest was written two seconds later by the + 5-minute capture job. Never over a complete primary, never after a failed restore, and **the capture + itself is not guarded** — it describes reality, and with the primary refilled there is nothing hollow + left to describe. **Per-app Tier-2 config panel (v0.57.0)** — `GET/POST /stacks/{name}/backup` (`internal/web/tier2_config_handler.go` + `templates/tier2_config.html`). The "2. mentés" row's