docs(v0.230.0): R-403 — CHANGELOG, CONTEXT rulings, README, REPORT
gates / gates (push) Failing after 13s
gates / gates (push) Failing after 13s
CHANGELOG v0.230.0, leading with the measurement rather than the fix: 120 082 104 B -> 7 036 B on the shipped v0.229.0, reproduced before anything was built. CONTEXT records three rulings: hollowness is a MANIFEST question and never a size question; the guard fences one shape and NOT shrinking, because the derived-copy rebuild is a design decision; and the rehydrate happens inside the restore because a follow-up job races the 5-minute capture. Plus the shape the live run taught: a warning that fires on everything costs the same as the comforting lie it replaces. README documents the refusal, what each surface says, and why the capture job is deliberately not guarded. REPORT leads with Part 1's result, carries the six red-proofs, the per-row Scenario D table with its seven-app control, and eight observations including R-404 filed-not-acted-on and three mistakes of mine recorded rather than tidied away.
This commit is contained in:
@@ -1,3 +1,97 @@
|
||||
## v0.230.0 — a poorer copy must never delete a richer one (2026-08-31, R-403)
|
||||
**MinAgent: 0.129.0** (unchanged)
|
||||
|
||||
### The loss was MEASURED first, then fixed
|
||||
|
||||
R-403 was filed yesterday from a code reading with the dangerous half explicitly recorded as
|
||||
**unverified**. It was run before anything was built, on the shipped **v0.229.0**, on `demo-hp`:
|
||||
|
||||
```
|
||||
BEFORE secondary db-dumps: 4 volume-dumps: 3 size: 120082104 bytes
|
||||
(a hollow primary unit, produced through the R-102 restore path exactly as on 2026-08-31)
|
||||
RUN [backup] Tier 2 copied docmost → …/backups/secondary/docmost (14.9 KB, 0 leg(s), 0s)
|
||||
AFTER secondary db-dumps: 0 volume-dumps: 0 size: 7036 bytes
|
||||
```
|
||||
|
||||
**Four database dumps and three volume tars — the customer's last surviving package — deleted in one
|
||||
nightly run, which recorded itself a SUCCESS.** Evidence:
|
||||
`felhom.eu/documentation/audits/DRILL-r403-tier2-delete-2026-08-31/`.
|
||||
|
||||
The mechanism was three individually-correct lines: `RunTier2` guarded the unit leg with `os.Stat`
|
||||
alone (*does the folder exist*), `rsyncMirror` is `rsync -a --delete`, and nothing between them
|
||||
compared source to destination. **An empty recovery unit is a folder that exists.**
|
||||
|
||||
### The guard
|
||||
|
||||
- **One predicate, `unitCarriesData` / `unitIsHollow`** (`internal/backup/r403_hollow.go`). It **asks
|
||||
the MANIFEST and never the byte size**: a unit with a big compose tree and no dumps is dangerous, a
|
||||
tiny unit for a tiny app is fine. Fail-closed on an absent or unparseable manifest.
|
||||
`TestR403_SizeIsNeverConsulted` is the guard that keeps `dirSizeBytes` out.
|
||||
- **`RunTier2` skips the unit leg** when the source is hollow **and** the destination is not. The
|
||||
other legs still run, the run is **not** failed, and the skip is recorded **for the surface**
|
||||
(`CrossDriveBackup.UnitLegSkipped` + `UnitPackageDate`) as well as logged loudly.
|
||||
- **`--delete` STAYS and shrinking stays legal.** `07-backup-architecture.md` §8 row 5's derived-copy
|
||||
rule ("Migration = rebuild, not preserve") is unchanged, and `tier2.go`'s own header records that a
|
||||
classified app's copy legitimately shrinks as `export` drops out. The fence is exactly one shape.
|
||||
`TestR403_DataLegShrinkIsUnaffected` is the guard on the guard.
|
||||
|
||||
### The honesty — a preserved copy must not read as a fresh one
|
||||
|
||||
A preserved package is older than the run that preserved it. The per-app card carries a notice, and
|
||||
the unit-restore confirm names the **package's** date — read from the mirrored manifest's own
|
||||
`created_at`, not from the status record — plus a clause saying why it is older. **The outcome
|
||||
sentence names the same date**, because §2.3's rule is "not a plain green success anywhere".
|
||||
|
||||
> **The live run caught a defect the unit tests did not.** The first draft also flagged "older" by
|
||||
> comparing the package's date to the run's — but a unit is ALWAYS captured shortly before the run
|
||||
> that mirrors it, so that was true for **every healthy app on the box**. Measured: bookstack, kimai,
|
||||
> opengist and privatebin all had manifests at `12:03:49Z` against a run at `12:14:24Z`, and all four
|
||||
> would have been told their package was stale. **A warning that fires on everything costs the same
|
||||
> as the comforting lie it replaces.** The flag is now `UnitLegPreserved` and nothing else;
|
||||
> `TestR403_AHealthyAppIsNeverCalledStale` pins it.
|
||||
|
||||
### The cause — the primary is filled back in
|
||||
|
||||
`RestoreTier2Unit` now refills an **absent or hollow** primary unit from the mirror it just restored
|
||||
from, **inside the call, before it returns**. The hollow manifest was written **two seconds** after a
|
||||
restore by the 5-minute capture job; any follow-up job or goroutine races it.
|
||||
`TestR403_RehydrateHappensBeforeTheCallReturns` asserts the ordering, never a timer.
|
||||
|
||||
Never over a **complete** primary (it may be newer — that is R-403 pointed the other way) and never
|
||||
after a **failed** restore. **The capture job is deliberately NOT guarded:** a capture that describes
|
||||
an empty drive as empty is correct, and with the primary refilled there is no hollow state left to
|
||||
describe. Guarding it would make the manifest lie.
|
||||
|
||||
### The rider — a credential reader that ends a three-time mistake
|
||||
|
||||
`felhom.eu/scripts/read_credential.py`. Values in `~/.config/credentials` are single-quoted; stripping
|
||||
only `"` sends two extra characters and the failure looks exactly like a stale password. **2026-07-20**
|
||||
it was diagnosed as drift and written into memory; **2026-08-31** it recurred and was caught;
|
||||
**2026-08-31, hours later, it recurred again and rewrote a live box's `password_hash`.** Between them
|
||||
the project already had a memory file, a worked recipe and a session report — none of it stopped
|
||||
occurrence three. The rule now lives in the code path: one matching quote pair is unwrapped, the
|
||||
result is **refused** if it still carries a quote, `--expect-length` gives the caller a second
|
||||
opinion, and the value goes file→file at 0600 with only its length on stdout.
|
||||
|
||||
### Live validation on `demo-hp`
|
||||
|
||||
| Scenario | Result |
|
||||
|---|---|
|
||||
| **A** — the loss on v0.229.0 | **CONFIRMED** — 120 082 104 B → 7 036 B |
|
||||
| **B** — the same state on v0.230.0 | **PRESERVED** — all 7 files, all sha256 identical, WARN quoted |
|
||||
| **C1** — both sides complete | mirrors as before (`114.5 MB, 0 leg(s)`, no skip) |
|
||||
| **D** — the surfaces | only the skipped app carries the notice; the other **seven are the control** |
|
||||
| **E** — the rehydrate | primary real the instant the call returned, and still real after **3** capture cycles |
|
||||
|
||||
### Tests
|
||||
|
||||
**24 new Go tests (1632 → 1656)** plus 2 Python tests. Red-proofs run and reverted: **A6** (predicate →
|
||||
size threshold), **B1** (guard removed → the copy's 3 files are DELETED and the seam is called), **B6**
|
||||
(a general never-shrink rule → the shrink case fails), **C2** (only-when-hollow dropped → the complete
|
||||
primary is overwritten), **E1** (quote assertion removed → three cases fail by name), plus the
|
||||
over-eager-stale red-proof. `recordTier2Success` and `tier2UnitConfirmMsg` keep their old signatures as
|
||||
thin callers, so **no existing test needed editing**.
|
||||
|
||||
## v0.229.0 — the second drive's copy becomes a way back (2026-08-31, R-102 + R-103)
|
||||
**MinAgent: 0.129.0** (unchanged)
|
||||
|
||||
|
||||
+41
-1
@@ -7,7 +7,47 @@
|
||||
>
|
||||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||||
|
||||
Last updated: 2026-08-31 (v0.229.0 — R-102/R-103: the second drive's copy becomes a way back)
|
||||
Last updated: 2026-08-31 (v0.230.0 — R-403: a poorer copy must never delete a richer one)
|
||||
|
||||
> **2026-08-31 — v0.230.0. THREE RULINGS, recorded so none is re-litigated.**
|
||||
>
|
||||
> **1. HOLLOWNESS IS A MANIFEST QUESTION, NEVER A SIZE QUESTION.** `unitCarriesData` asks whether the
|
||||
> unit's manifest lists any database dump or any volume tar, and nothing else. `dirSizeBytes` lives
|
||||
> two files away and is the obvious wrong answer: a unit with a fat compose capture and no dumps is
|
||||
> exactly the shape that deleted 120 MB on `demo-hp`, and a 360-byte unit belonging to a tiny app is
|
||||
> perfectly healthy. Size answers *how big*; the question is *is there anything to recover*. Absent or
|
||||
> unparseable manifest ⇒ hollow, fail closed: a unit whose contents cannot be vouched for must never
|
||||
> authorise a delete of one whose contents can. `TestR403_SizeIsNeverConsulted` is the fence.
|
||||
>
|
||||
> **2. THE GUARD FENCES ONE SHAPE, NOT SHRINKING — because the derived-copy rebuild is a DESIGN
|
||||
> DECISION.** `07-backup-architecture.md` §8 row 5 records that the secondary is a derived copy,
|
||||
> rebuilt on the next run, and `tier2.go`'s own header records that a classified app's copy
|
||||
> legitimately shrinks as `export` drops out of its class set. `rsync -a --delete` stays, the data legs
|
||||
> are untouched, and complete→hollow and hollow→hollow both still mirror. The ONLY refusal is a source
|
||||
> carrying no data over a destination that carries some. Widening this to "the secondary never shrinks"
|
||||
> would be calling a decision a defect; `TestR403_DataLegShrinkIsUnaffected` is the guard on the guard.
|
||||
>
|
||||
> **3. THE REHYDRATE HAPPENS INSIDE THE RESTORE, BECAUSE A FOLLOW-UP JOB RACES THE CAPTURE.** The
|
||||
> hollow manifest was written **two seconds** after a Tier-2 unit restore, by the 5-minute
|
||||
> `backup-cache` job. A goroutine, a scheduled refresh or a "do it on the next run" would each lose
|
||||
> that race some of the time, and the failure mode is silent. `RestoreTier2Unit` refills the primary
|
||||
> before it returns, and the test asserts ORDERING rather than sleeping.
|
||||
> **And the capture is deliberately NOT guarded:** a capture that describes an empty drive as empty is
|
||||
> CORRECT. With the primary refilled there is no hollow state left to describe. Guarding the capture
|
||||
> would have made the manifest lie, which is the opposite of every other fix this week.
|
||||
>
|
||||
> **A defect the LIVE run caught and the unit tests did not, worth remembering as a shape.** The first
|
||||
> draft of `UnitRestoreDate` also flagged "the package is older than the run" by comparing their dates
|
||||
> — and a unit is ALWAYS captured shortly before the run that mirrors it, so it was true for every
|
||||
> healthy app on the box. Four apps would have been told their package was stale. **A warning that
|
||||
> fires on everything costs the same as the comforting lie it replaces.** The flag is now
|
||||
> `UnitLegPreserved` and nothing else.
|
||||
>
|
||||
> **The credential rider.** `felhom.eu/scripts/read_credential.py` is now the one place that value is
|
||||
> read. Three occurrences (2026-07-20, and twice on 2026-08-31, the third of which rewrote a live box's
|
||||
> password hash) happened while the project already had a memory file, a worked recipe and a session
|
||||
> report describing the mistake. **A note is read by whoever thinks to look; a check runs whether or
|
||||
> not anyone remembers.**
|
||||
|
||||
> **2026-08-31 — v0.229.0. THREE RULINGS, recorded so none is re-litigated.**
|
||||
>
|
||||
|
||||
@@ -1,322 +1,278 @@
|
||||
# REPORT — R-102 + R-103: the second drive's copy becomes a way back
|
||||
# REPORT — R-403: a poorer copy must never delete a richer one
|
||||
|
||||
**Controller v0.229.0 · 2026-08-31 · MinAgent 0.129.0 (unchanged)**
|
||||
**Controller v0.230.0 · 2026-08-31 · MinAgent 0.129.0 (unchanged)**
|
||||
|
||||
---
|
||||
|
||||
## 2. PART 1'S RESULT, FIRST — the loss is REAL and was reproduced before anything was built
|
||||
|
||||
**On the shipped v0.229.0, on `demo-hp`, app `docmost`.** The hollow primary was produced through the
|
||||
**R-102 restore path**, exactly as the 2026-08-31 observation was — not hand-crafted.
|
||||
|
||||
```
|
||||
BEFORE AFTER (one Tier-2 run)
|
||||
db-dumps : 4 files db-dumps : 0 files
|
||||
volume-dumps : 3 files volume-dumps : 0 files
|
||||
unit size : 120 082 104 bytes unit size : 7 036 bytes
|
||||
|
||||
9f676376…a092a28 db-dumps/docmost-postgres.sql GONE
|
||||
73917ba6…fe15ef1 db-dumps/pre-restore-20260822T162347Z-…sql GONE
|
||||
4c134c2c…4935949 db-dumps/pre-restore-20260822T162708Z-…sql GONE
|
||||
13e5a864…422af25 db-dumps/pre-restore-20260822T215432Z-…sql GONE
|
||||
f46a2fc3…4c8e3b1 volume-dumps/docmost_docmost_postgres_data.tar GONE
|
||||
a8df17c4…1cfa1a73 volume-dumps/docmost_docmost_redis_data.tar GONE
|
||||
88f21f49…5f2ba751d volume-dumps/docmost_docmost_storage.tar GONE
|
||||
```
|
||||
|
||||
The run's own line: `[backup] Tier 2 copied docmost → …/backups/secondary/docmost (14.9 KB, 0 leg(s),
|
||||
0s)` — **recorded as a success.**
|
||||
|
||||
**VERDICT: LOSS CONFIRMED.** The code reading filed yesterday was right, and it is now a measurement.
|
||||
The hollow primary manifest that armed it: `created_at 2026-08-31T11:32:47Z`, `db_dumps: []`,
|
||||
`volume_dumps: null`, written by the 5-minute `backup-cache` job ~140 s after the restore.
|
||||
|
||||
Evidence: `felhom.eu/documentation/audits/DRILL-r403-tier2-delete-2026-08-31/` — `phase1a` (before),
|
||||
`phase1b` (the hollow primary), `phase1c` (the loss), `phase1d` (repair).
|
||||
|
||||
## 1. Confirmed baselines
|
||||
|
||||
Re-checked before the first edit, and both matched the task's table exactly. **No drift.**
|
||||
Re-checked before the first edit. **No drift.**
|
||||
|
||||
| Repo | `main` @ start | expected | version |
|
||||
|---|---|---|---|
|
||||
| `felhom-controller` | `430fb4448d6175064ef4e86c2b3f8796e15ae30c` | same | `v0.228.0` → **`v0.229.0`** |
|
||||
| `felhom.eu` | `1623a4d5b5d5d0f1b73aba2727b8de053930f336` | same | — (docs only) |
|
||||
| `felhom-controller` | `fed272e62adf3c72a2c61f7f2f1f0a5e21164c71` | same | `v0.229.0` → **`v0.230.0`** |
|
||||
| `felhom.eu` | `83ff9e8e3856fa822bfa1f80fc305783964aef68` | same | — (docs + one script) |
|
||||
| `felhom-agent` | not touched | — | `v0.130.0` unchanged |
|
||||
| `app-catalog-felhom.eu` | `459766cb16395fd1d1a66282f5cc6da59ead5924` | read-only | unchanged |
|
||||
|
||||
`git status --porcelain` empty in both repos; disk 37% / 51%.
|
||||
`git status --porcelain` empty in both; disk 37% / 51%; `demo-hp` was running the shipped 0.229.0,
|
||||
which is what Part 1 required.
|
||||
|
||||
## 2. Files created / modified
|
||||
## 3. Files created / modified
|
||||
|
||||
**`felhom-controller`**
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `controller/internal/appbackup/paths.go` | +4 unit-directory-relative primitives; the 4 `(nsRoot, stackName)` helpers become wrappers |
|
||||
| `controller/internal/appbackup/r102_paths_split_test.go` | **new** — A1 + a directory-relative guard |
|
||||
| `controller/internal/backup/appbackup_bridge.go` | re-exports the 4 primitives into `backup` |
|
||||
| `controller/internal/backup/restore_unit.go` | `RestoreFromRecoveryUnitAt(stack, unitDir)`; the 1-arg form becomes the thin caller; the volume leg goes through the R-354 seam; the unit dir is logged |
|
||||
| `controller/internal/backup/restore_db.go` | `reimportDBDumpsAtCtx`; `dbReimportTimeout` named once |
|
||||
| `controller/internal/backup/tier2_restore.go` | `RestoreTier2Unit`, `ErrTier2NoUnitInCopy`, `tier2UnitDir`, `tier2UnitIsOpenable`, `Tier2Coverage.{UnitRestorable,CopyLastRun,CopyLastSuccess}`, `CanRestoreUnit()`, `Tier2CopyDate()` |
|
||||
| `controller/internal/backup/r102_unit_at_test.go` | **new** — A2…A6 + 1 |
|
||||
| `controller/internal/backup/r102_tier2_unit_test.go` | **new** — B1…B5 + 1 |
|
||||
| `controller/internal/backup/r103_file_restore_untouched_test.go` | **new** — C1, C2 |
|
||||
| `controller/internal/web/handlers.go` | `backupTier2UnitRestoreHandler`; the refusal split; 7 named constants; `tier2UnitConfirmMsg`, `tier2UnitSourceMsg`; 4 new `AppBackupRow` fields + their builder |
|
||||
| `controller/internal/web/server.go` | route `POST /backup/tier2/unit-restore` |
|
||||
| `controller/internal/web/funcmap.go` | `fmtTimeStr` delegates to a package-level `fmtRFC3339Local` |
|
||||
| `controller/internal/web/templates/backups_apps.html` | the destructive action on the Tier-2 row |
|
||||
| `controller/internal/web/r103_tier2_action_test.go` | **new** — D1…D6 + 4 |
|
||||
| `CHANGELOG.md`, `CONTEXT.md`, `controller/README.md`, `REUSE.md`, `REPORT.md` | documentation |
|
||||
| `controller/internal/backup/r403_hollow.go` | **new** — `unitCarriesData` / `unitIsHollow`, the ONE predicate |
|
||||
| `controller/internal/backup/tier2.go` | the unit-leg precondition; `tier2UnitPreservedWarning`; `unitPackageDate`; `recordTier2SuccessWithUnit` (the 5-arg form kept as a thin caller) |
|
||||
| `controller/internal/backup/tier2_shares.go` | one comment — shares have no unit leg |
|
||||
| `controller/internal/backup/tier2_restore.go` | `rehydratePrimaryUnit`, `countUnitFiles`; `Tier2Coverage.{UnitPackageDate,UnitLegPreserved}`; `UnitRestoreDate()` |
|
||||
| `controller/internal/backup/backup.go` | the `unitRehydrate` seam |
|
||||
| `controller/internal/settings/settings.go` | `CrossDriveBackup.{UnitLegSkipped,UnitPackageDate}` |
|
||||
| `controller/internal/web/handlers.go` | `tier2UnitStaleClause`, `tier2UnitStaleNoticeFmt`, `tier2UnitConfirmWithStaleness` (2-arg form kept as a thin caller); the row's `Tier2UnitStaleNotice`; `tier2UnitSourceMsg` now names the package date |
|
||||
| `controller/internal/web/templates/backups_apps.html` | the stale notice on the card |
|
||||
| `controller/internal/backup/r403_{hollow,mirror_guard,rehydrate}_test.go`, `controller/internal/web/r403_surface_test.go` | **new** — Groups A–D |
|
||||
| `CHANGELOG.md`, `CONTEXT.md`, `controller/README.md`, `REPORT.md` | documentation |
|
||||
|
||||
**`felhom.eu`** — `documentation/architecture/07-backup-architecture.md` (§6.2, §6.3, §7.2, §8, §8.1),
|
||||
`documentation/architecture/00-capability-map.md` (header note, Tier-2 row, D5 row),
|
||||
`documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/CLOSED-ITEMS.md`, `STATUS.md`,
|
||||
`documentation/audits/DRILL-r102-tier2-unit-2026-08-31/` (**new**, 11 files).
|
||||
**`felhom.eu`** — `scripts/read_credential.py` + `scripts/test_read_credential.py` (**new**, Part 4);
|
||||
`documentation/architecture/07-backup-architecture.md` (§8 row 5 note + new **§8.2**);
|
||||
`documentation/architecture/00-capability-map.md`; `documentation/backlog/{OPEN,CLOSED}-ITEMS.md`;
|
||||
`STATUS.md`; `documentation/audits/DRILL-r403-tier2-delete-2026-08-31/` (**new**, 9 files).
|
||||
|
||||
**Not modified:** `felhom-agent`, `app-catalog-felhom.eu` (Part 3 is a measurement), `ROADMAP.md`
|
||||
(it carries no R-102/R-103 row — `one_register_gate.py` green).
|
||||
**Not modified:** `felhom-agent`, `app-catalog-felhom.eu`, `rsyncMirror`, the data legs, the R-165/R-181
|
||||
capture floor, the off-site path, `golden_currency_gate.py` (that is R-404 and it is Viktor's).
|
||||
|
||||
## 3. Commits pushed to `main`
|
||||
## 4. Commits pushed to `main`
|
||||
|
||||
| Repo | Commit | What |
|
||||
|---|---|---|
|
||||
| `felhom-controller` | `c732006d263a25744eca11a11cdeef343b1c95af` | Part 1.1 — path helper split, zero behaviour change |
|
||||
| `felhom-controller` | `0f9b796615d3e662fc010b747fb5048f36faffb7` | R-102 — Parts 1.2, 1.3, 2.1 + Groups A and B |
|
||||
| `felhom-controller` | `4c8f0d291948fd60887c0fd7066e63df9e3abb68` | R-103 — Part 2.2 + Groups C and D |
|
||||
| `felhom-controller` | `8aa95b58319f947fa9b3de38cb410b9d2756332f` | CHANGELOG / CONTEXT / README / REUSE / REPORT |
|
||||
| `felhom.eu` | `c2de785bf27631284fa5ab525e0f32494804773a` | architecture, register, STATUS, drill evidence — pushed with `--no-verify`, declared |
|
||||
| `felhom-controller` | `0957f43d108e4166c545e4fbb5e11481594ec218` | commit hashes recorded in §3 |
|
||||
| `felhom.eu` | `83ff9e8e3856fa822bfa1f80fc305783964aef68` | golden 0.229.0 bake record, R-242 paid, STATUS — pushed with the hook ARMED, no bypass |
|
||||
| `felhom-controller` | *(this update)* | §6 and §12.2 rewritten for the completed delivery |
|
||||
| `felhom.eu` | `66156c619fd2bceff5af6213f11a0836183f6880` | drill evidence + `read_credential.py` |
|
||||
| `felhom-controller` | `2358e561b741b3f08e254b28b29d8b6906dd52a0` | the guard, the honesty, the rehydrate, Groups A–D |
|
||||
| `felhom-controller` | `5429d651ee882ce3354216aa44f4669dce15e30b` | the over-eager stale flag, caught by the live run |
|
||||
| `felhom-controller` | `b48a7fa326dbe5505d1568ac55c66488e8de125c` | the outcome names the package date |
|
||||
| `felhom-controller` | *(docs commit — §14)* | CHANGELOG / CONTEXT / README / REPORT |
|
||||
| `felhom.eu` | *(docs commit — §14)* | architecture §8.2, capability map, register, STATUS |
|
||||
|
||||
## 4. Per-test results, and the five named red-proofs
|
||||
## 5. Per-test results and the named red-proofs
|
||||
|
||||
Full suite: `go build ./... && go vet ./... && go test ./...` — **all packages ok**
|
||||
(`internal/backup` 296.0 s, the rest cached/fast). `python3 controller/scripts/controller_gates.py` —
|
||||
**all 13 gates OK**.
|
||||
(`internal/backup` 313 s). `controller_gates.py` — **all 13 OK**. `test_read_credential.py` — **OK**.
|
||||
|
||||
| Group | Tests | Result |
|
||||
|---|---|---|
|
||||
| A (`appbackup`, `backup`) | `TestR102_PathWrappersAreByteIdenticalToToday`, `…UnitHelpersAreDirectoryRelative`, `…RestoreFromRecoveryUnitAtReadsTheGivenDir`, `…PrimaryPathIsUnchanged`, `…LiveDestinationIsUnchanged`, `…MutationOrderIsPreserved`, `…MissingManifestInMirrorFailsClosed` (3 sub-cases), `…UnopenableMirrorIsStillDisclosedAsUnread` | PASS |
|
||||
| B (`backup`) | `TestR102_Tier2UnitRestoreReadsTheSecondaryMirror`, `…WorksWithThePrimaryUnitABSENT`, `…TakesTheSingleWriterFlag`, `TestR102_NoTier2CopyIsAnHonestRefusal`, `…CopyAgeIsCarriedToTheSurface`, `…DoesNotWriteTheMirror` | PASS |
|
||||
| C (`backup`) | `TestR103_CanRestoreStillAnswersLegsOnly`, `TestR103_FileRestoreBehaviourUnchanged` | PASS |
|
||||
| D (`web`) | `TestR103_UnitActionOfferedWhenTheMirrorExists`, `…AbsentWhenItDoesNot`, `…ConfirmStatesTheOverwrite`, `…ConfirmNamesTheCopyDate`, `…AvailableMsgNamesTheButtonByItsLabel`, `…SecondPressIsRefused`, `…HandlerPublishesTheOutcome`, `…RefusesAnUnopenableMirrorWithoutStopping`, `…UnitRestoreHandlerGuards`, `…FileRestoreRefusalPointsAtTheActionThatWorks` | PASS |
|
||||
| E | the existing suite unmodified — **no existing test was edited** | PASS |
|
||||
| A | `TestR403_{ManifestWithNoDumpsIsHollow, ManifestWithVolumeDumpsOnlyIsNotHollow, ManifestWithDBDumpsOnlyIsNotHollow, AbsentManifestIsHollow, UnparseableManifestIsHollow, SizeIsNeverConsulted, AHealthyAppIsNeverCalledStale}` | PASS |
|
||||
| B | `TestR403_{HollowSourceOverCompleteDestIsSkipped, CompleteSourceStillMirrors, HollowOverHollowStillMirrors, FirstCopyStillMirrors, OtherLegsStillRunWhenTheUnitLegIsSkipped, SkipIsRecordedForTheSurface, DataLegShrinkIsUnaffected, GuardUsesTheSharedPredicate}` | PASS |
|
||||
| C | `TestR403_{HollowPrimaryIsRefilledFromTheMirror, CompletePrimaryIsLeftByteIdentical, FailedRestoreDoesNotWriteAPackage, RehydrateHappensBeforeTheCallReturns, RehydrateFailureDoesNotFailTheRestore}` | PASS |
|
||||
| D | `TestR403_{SkippedUnitLegIsNotRenderedAsFresh, UnitRestoreOfferNamesTheOlderPackageDate, TheOrdinaryConfirmIsUnchanged, OutcomeNamesThePackageDateNotTheRunDate}` | PASS |
|
||||
| E | `test_r404_credential_length_mismatch_fails_loudly`, `test_the_value_is_never_printed` | PASS |
|
||||
| E2 | the existing suite unmodified — **no existing test was edited** (both changed signatures kept their old form as thin callers) | PASS |
|
||||
|
||||
**Red-proofs — each mutated, run, observed failing, reverted:**
|
||||
|
||||
| # | Mutation | Observed failure |
|
||||
|---|---|---|
|
||||
| **A1** | `UnitComposeDir` joins `"compose2"` | FAIL on all three fixtures: `…/docmost/compose2, want …/docmost/compose` |
|
||||
| **A5** | recreate the definition **before** replaying the volumes | FAIL: *"volumes were replayed AFTER the definition was recreated"* + *"the volume replay had not run when the definition was recreated"* |
|
||||
| **B2** | `RestoreTier2Unit` points back at the primary unit path | FAIL twice: `config came from the PRIMARY unit: SUBDOMAIN="primary"`, and with the primary tree unreadable, `reading volume dump dir: … permission denied` |
|
||||
| **C1** | `CanRestore()` widened to `len(Legs) > 0 \|\| HasUnit` | FAIL on both unit-only cases |
|
||||
| **D6** | drop `EndRestoreOp` from the handler's goroutine | FAIL after 30 s: *"the restore never published a result"* |
|
||||
| **A6** | predicate → `dirSizeBytes > 1024` | FAIL: *"a 400346-byte unit listing NO dumps was called data-bearing"* + *"a 360-byte unit listing a volume tar was called hollow"* |
|
||||
| **B1** | the guard removed | FAIL: *"the destination unit CHANGED"*, all three files *"was DELETED from the copy"*, and *"the mirror seam WAS called for the unit leg"* |
|
||||
| **B6** | a general never-shrink rule (refuse any leg whose destination exists) | FAIL: *"a data leg stopped shrinking — the guard is TOO WIDE and is fencing a design decision"* |
|
||||
| **C2** | the only-when-hollow condition dropped | FAIL: *"the rehydrate ran 1 time(s) over a COMPLETE primary"* |
|
||||
| **E1** | the quote assertion removed | FAIL ×3 by name: one-sided strip, trailing-only quote, mismatched pair |
|
||||
| *(extra)* | reinstate the package-older-than-the-run comparison | FAIL: *"a healthy app … was flagged as preserved/stale"* |
|
||||
|
||||
## 5. Test count
|
||||
> **B6's first mutation was wrong and is recorded rather than quietly re-done.** It skipped the data
|
||||
> legs only when the unit leg was skipped, and B6's fixture has a COMPLETE source, so the mutation
|
||||
> never reached it — `OtherLegsStillRun` failed instead. Re-done as a true never-shrink rule, which is
|
||||
> what B6 actually guards, and then it failed correctly.
|
||||
|
||||
**1606 → 1632 test functions (+26)**, counted as unique `^func Test…` across `controller/**/*_test.go`
|
||||
at `430fb44` and at HEAD.
|
||||
## 6. Test count
|
||||
|
||||
## 6. Deployed version
|
||||
**Go: 1632 → 1656 (+24).** Python: +2 (`test_read_credential.py`).
|
||||
|
||||
## 7. Deployed version
|
||||
|
||||
```
|
||||
$ ssh hp "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.229.0 Up 22 seconds (healthy)
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.230.0 Up (healthy)
|
||||
```
|
||||
|
||||
Built locally (`build.sh 0.229.0 --push`) from a clean, pushed tree; deployed to demo-hp guest 9201 by
|
||||
the bootstrap mechanism (`docker pull` → `/etc/felhom-controller-image` → restart the unit).
|
||||
**The fleet is on 0.229.0 — WHICH CARRIES THE DEFECT.** `demo-hp` was updated by hand; `demo-felhom`
|
||||
is still on 0.229.0. **A golden carrying 0.230.0 is owed** (R-242, Viktor's), and this time the day-0
|
||||
ground that justified the previous six bypasses **does not apply**: R-403 is a defect in the nightly
|
||||
Tier-2 copy, which a newly installed box starts running on its first night.
|
||||
|
||||
**Fleet delivery is DONE** — on the operator's explicit instruction, given after the report above was
|
||||
first written. Golden **0.229.0** baked, published, **round-trip verified** (656 864 331 B, sha256
|
||||
`39aa886d…d7bdae87`, both identical to what the bake reported; `./etc/felhom-controller-image` read out
|
||||
of the *downloaded* archive says `felhom-controller:0.229.0`), **vouched** as a three-field change
|
||||
(`golden_version` 0.229.0 · `agent_version` 0.130.0 · `min_agent` 0.129.0, the last read from this
|
||||
CHANGELOG's header rather than assumed), and the **fleet floor raised to 0.229.0**. Verified by
|
||||
re-reading the manifest, not by trusting the flash. **The floor is proven ACTING:** `demo-felhom`
|
||||
self-updated `0.228.0 → 0.229.0` with nobody deploying to it. `golden_currency_gate.py` went red →
|
||||
green. Evidence: `felhom.eu/documentation/tests/golden-0.229.0-2026-08-31/`.
|
||||
## 8. The live evidence — Scenarios B, D, E
|
||||
|
||||
## 7. NOT yet live-validated — per matrix row, deliberately
|
||||
**Scenario B — the same state, on the fixed build.** The WARN, verbatim:
|
||||
|
||||
- **§8 row 3b — PROVEN.** The route was exercised live end-to-end with the primary unit absent, and the
|
||||
verdict is taken from the DATA, not from an exit code.
|
||||
- **§8 row 4 — stays PARTIAL, and this is the sentence that must not be overstated.** What was proven is
|
||||
the ROUTE, not the JOURNEY. **No drive has ever actually died or been replaced under this recovery.**
|
||||
The drill removed a *unit directory*; it did not remove a *disk*. So drive re-attachment by
|
||||
`durable_id`, the agent's enrolment of a replacement, and a Tier-2 copy read from a drive that is the
|
||||
only surviving one are all still unexercised. Promoting row 4 needs that journey.
|
||||
- **Not exercised at all:** the Tier-2 copy on **network storage** (§8 of the task's edge table — the
|
||||
behaviour is inherited unchanged and was not re-decided, so it was not re-tested); the
|
||||
**primary-newer-than-the-mirror** case (both dates are stated and no steering logic was built, per
|
||||
§12); and the **R-403 consequence** below.
|
||||
- **Rendering is unproven, as always here.** `claude-in-chrome` is unavailable on DooPlex, so the drill
|
||||
was endpoint-level: the exact routes the buttons post to, with a real session cookie and a real
|
||||
session CSRF. The confirm string was read out of the **rendered page**, so the markup is proven; a
|
||||
human click-through is still the only proof of the browser dialog.
|
||||
```
|
||||
[WARN] [backup] Tier 2 docmost: unit leg SKIPPED — the recovery unit on the source drive lists no
|
||||
database dumps and no volume tars, while the existing copy at
|
||||
/mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit does. The copy was PRESERVED rather
|
||||
than replaced with an empty one (R-403). The other legs continue.
|
||||
[INFO] [backup] Tier 2 copied docmost → …/secondary/docmost (14.9 KB, 0 leg(s), 0s)
|
||||
[unit leg SKIPPED — existing package preserved, R-403]
|
||||
```
|
||||
|
||||
## 8. The drill, step by step
|
||||
`db-dumps: 4 volume-dumps: 3 size: 120082104` **before and after**, and all **seven sha256 values
|
||||
identical**. On v0.229.0 the same state left 0 files.
|
||||
|
||||
Evidence: `felhom.eu/documentation/audits/DRILL-r102-tier2-unit-2026-08-31/` (README + 9 phase logs +
|
||||
the hollow manifest). Machine: **demo-hp**, guest 9201 — Tier 0, disposable. `demo-felhom`, `ep0`,
|
||||
DooPlex and Peti's box untouched. App: **docmost**, class B — its own Tier-2 run reports **`0 leg(s)`**.
|
||||
**Scenario D — the surfaces, per row, with the other seven apps as the control:**
|
||||
|
||||
1. **The mirror's contents were confirmed, not assumed:** manifest schema 2, `compose/{app.yaml,
|
||||
docker-compose.yml,.felhom.yml}`, 3 volume tars, 1 canonical `.sql` (+3 `pre-restore-` undo copies).
|
||||
2. **Observables planted.** A file with an accented Hungarian name in docmost's own storage root
|
||||
(`/app/data/storage`), and a row in docmost's own Postgres via docmost's own DB role.
|
||||
`content sha256 9228fddade66a054…c444`; `name utf-8 hex
|
||||
c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e747874` (35 bytes).
|
||||
3. **Captured and mirrored** through `POST /api/backup/run` then `POST /api/backup/tier2`. Primary and
|
||||
mirror then **byte-identical on all four artefacts** (sha256 printed for each).
|
||||
4. **A post-backup discriminator added** — `csak-mentes-utan.txt` and a `post-backup` row — so a
|
||||
restore that changed nothing could not pass as a restore that worked.
|
||||
5. **The live data destroyed, and the loss proven BY THE OBSERVABLE.** *The first attempt destroyed
|
||||
nothing:* `docker volume rm` was refused because the stopped containers still referenced the
|
||||
volumes, and printed nothing. **Recorded at the top of `phase4-destroy.log` rather than quietly
|
||||
re-run** — an unchecked exit code that looks like success is the trap this project has a standing
|
||||
rule about. Re-done as an in-place wipe: 68 989 735 B → **0**, and the app's own database then
|
||||
answered `ERROR: relation "felhom_r102_discriminator" does not exist`.
|
||||
6. **The PRIMARY unit moved aside** — `mv backups/primary/docmost backups/primary/docmost.ASIDE-r102`;
|
||||
`ls` on the original path returns `No such file or directory`. **Without this the drill proves only
|
||||
that the code runs.**
|
||||
7. **Restored through the real endpoint** — `POST /backup/tier2/unit-restore`, session + 64-char CSRF.
|
||||
`302` → *„Teljes visszaállítás elindult"*. The controller's own line names the source:
|
||||
`Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit:
|
||||
images=3, secrets recovered=2/2, data_keys=0` → `3 volume(s) of 3 listed, 1 database(s) of 1 listed`,
|
||||
**28.65 s**. The published outcome: *„A(z) docmost: 3 adatkötet és az adatbázis visszaállítva …
|
||||
A visszaállítás forrása a második meghajtón lévő másolat volt (2026-08-31 12:00)."*
|
||||
8. **Proven byte-for-byte, and proven THROUGH THE APP.** The accented name came back with the identical
|
||||
35-byte hex above (compared as **hex**, never as rendered text — R-364), content sha256 identical;
|
||||
the post-backup file and the post-backup row were **GONE**; a psql client on docmost's own network
|
||||
with docmost's own credential returned `docmost@172.20.0.2/32:5432` and the `pre-backup` row; 44
|
||||
tables present; docmost answered **HTTP 200**.
|
||||
9. **Scenario D.** Repeated with the guest's `app.yaml` moved aside and a fresh `phase8-marker` row
|
||||
planted first. Result: `secrets recovered=2/2` **from the mirrored unit**, the guest's `app.yaml`
|
||||
rebuilt from it at 0600, `phase8-marker` **gone**, the accented file byte-identical, HTTP 200.
|
||||
**This closes `00-capability-map.md`'s open clause** for Tier-2's own cross-drive copy of a
|
||||
secret-bearing unit.
|
||||
10. **The primary put back, and the ordinary path re-proved** — `3 volume(s) of 3, 1 database(s) of 1`
|
||||
from `…/backups/primary/docmost`, accented file byte-identical, HTTP 200.
|
||||
```
|
||||
app notice FIGYELEM package date in the confirm
|
||||
bookstack False False 2026-08-31 14:03
|
||||
calibre-web False False 2026-08-31 14:03
|
||||
docmost True True 2026-08-31 11:43 <- the preserved package
|
||||
kimai False False 2026-08-31 14:03
|
||||
opengist False False 2026-08-31 14:03
|
||||
paperless-ngx False False 2026-08-31 14:03
|
||||
privatebin False False 2026-08-31 14:03
|
||||
romm False False 2026-08-31 14:03
|
||||
```
|
||||
|
||||
**Evidence was copied off the box at the end of each phase, before the reverts** (R-320): every phase
|
||||
log was `tee`'d to DooPlex as it ran, and the hollow manifest was pulled off **before** it was deleted.
|
||||
Only the skipped app carries the notice, and its confirm names the **package's** date (11:43) while
|
||||
every other row names its freshly-mirrored one (14:03). ASCII fragments (`adatcsomagja`, `FIGYELEM`)
|
||||
with the seven other rows as the negative control.
|
||||
|
||||
## 9. Part 3 — the count
|
||||
**Scenario E — the rehydrate.** Immediately after the call returned, with nothing waited for:
|
||||
|
||||
**A = 7 · B = 45 · C = 1**, counted at catalogue **`459766cb16395fd1d1a66282f5cc6da59ead5924`**.
|
||||
```
|
||||
BEFORE created_at: 2026-08-31T12:08:49Z db_dumps: [] volume_dumps: None
|
||||
AFTER created_at: 2026-08-31T09:43:41Z db_dumps: ['docmost-postgres.sql']
|
||||
volume_dumps: ['…postgres_data.tar','…redis_data.tar','…storage.tar']
|
||||
volume tars on the app drive: 3 db dumps: 4
|
||||
[INFO] [backup] docmost: primary unit refilled from the secondary mirror (R-403) —
|
||||
3 volume tar(s), 4 database dump(s) now on the app's own drive
|
||||
```
|
||||
|
||||
**Rule applied:** the production pipeline, not a restatement of it — `stacks.LoadMetadata` (the single
|
||||
validation choke point, so a rejected `backup:` block degrades to legacy exactly as it does live) →
|
||||
`stacks.ParseComposeClassifiableBinds` → `appbackup.ClassifyBinds` → `appbackup.ComputeCaptureSet` at
|
||||
`TierSecondary`, with the legacy branch falling back to `AppDataBindsPresent` + `AppDataDirNames` as
|
||||
`backup.tier2CaptureSet` does. A = at least one leg survives that pipeline. 13 templates carry a valid
|
||||
`backup:` block; the other 40 are legacy and none binds a namespace path.
|
||||
And **after waiting out 3 `backup-cache` cycles** (330 s) — the job that wrote the hollow manifest in
|
||||
the first place — the primary is still a real package: `created_at 2026-08-31T12:28:32Z`, 1 db dump,
|
||||
3 volume tars. **The hollow state is gone, and the capture is describing reality.**
|
||||
|
||||
- **A (7):** audiobookshelf, calibre-web, immich, komga, nextcloud, paperless-ngx, romm.
|
||||
- **C (1):** bentopdf — no `volumes:` key and no `${…_PATH}` bind at all.
|
||||
- **B (45):** the rest.
|
||||
## 9. NOT live-validated — explicit
|
||||
|
||||
**How the disagreement arose — established, not guessed.** The INV Part B.1 count (7/45/1) was right.
|
||||
Phase 0's own write-up says four apps are in B *"only because their single bind is a `:ro` media mount,
|
||||
which `ClassifyBinds` correctly excludes"* — it applied the **`:ro` default** rule. The two it therefore
|
||||
missed are **radarr and sonarr**: their `${USERDATA_PATH}` binds are **writable**, so that rule never
|
||||
reaches them, and they are excluded by an **explicit `class: excluded`** entry instead. 9 − 2 = 7 and
|
||||
43 + 2 = 45 — exactly the gap. The counting tool was temporary and was deleted; the method above
|
||||
re-runs it. **No catalogue file was changed.**
|
||||
- **Scenario C3/C4 live** (hollow→hollow, and a data leg shrinking) — unit-tested only. The live box
|
||||
had no app in either state and manufacturing one would have meant breaking a second app's backup.
|
||||
- **The rehydrate's failure path** (`unitRehydrate` returning an error) — unit-tested only; no way to
|
||||
make a real `rsync` fail on that box without damaging something.
|
||||
- **A real second-drive failure.** Everything here was proven by making a package hollow, never by
|
||||
removing a disk. `07` §8 row 4 remains PARTIAL for that reason and did not move.
|
||||
- **`demo-felhom`** was deliberately untouched; the fix is proven on one machine.
|
||||
- **Rendering** — endpoint level, as always here: the markup is proven, the browser dialog is not.
|
||||
|
||||
## 10. Rows moved, and to what
|
||||
## 10. Rows moved
|
||||
|
||||
| Row | Was | Now | Citation |
|
||||
|---|---|---|---|
|
||||
| `07` §8 row **3b** | `NONE` — "no route" | **`PROVEN`**, RTO **28.65 s**, RPO 24 h | `audits/DRILL-r102-tier2-unit-2026-08-31/` |
|
||||
| `07` §8 row **4** | `PARTIAL` — "the Tier-2 copy's volume tars remain unreachable" | **`PARTIAL`, unchanged status, changed reason** — the unreachability is closed; the drive-loss JOURNEY is still unexercised | §7.2 + the same drill, with the limit stated per row |
|
||||
| `07` §8.1 blanks | 3b: RTO+RPO; 4: RTO | 3b **removed** (both measured); 4 kept, with the route/journey distinction spelled out | — |
|
||||
| `07` §6.3 Tier-2 row | "the unit mirror is read by nothing" | **CLOSED — R-102**, old sentence kept in the past tense per the section's own practice | drill |
|
||||
| `07` §7.2 first bullet | open | **CLOSED**, and it says plainly that Tier-2 can now meet its prerequisite in the failure it exists for | drill |
|
||||
| `07` §6.2 | ⚠ UNRESOLVED, two counts | **✔ RESOLVED — 7/45/1**, with which was wrong and why | catalogue `459766cb1639` |
|
||||
| `00-capability-map` Tier-2 row | "the mirror is read by no path (→ R-102)" | **R-102 CLOSED**, route named, §8 row 3b added | drill |
|
||||
| `00-capability-map` D5 row | *"Not exercised live: … Tier-2's own cross-drive copy of a secret-bearing unit"* | that half **struck**, with the evidence path | `phase8-scenarioD-…log` |
|
||||
| `00-capability-map` header | "two counts disagree … do not adopt either" | points at the settled number | `07` §6.2 |
|
||||
- **`07` §8 row 5** — status **unchanged**; a pointer added to the new **§8.2**, which states the
|
||||
derived-copy rule is intact and names the single exception, so a future reader does not "fix" the
|
||||
skip back.
|
||||
- **`07` §8.2** — **new section**, with the measurement, the four-case table, and the reason the data
|
||||
legs are not guarded.
|
||||
- **`00-capability-map.md` Tier-2 row** — **no status change, stated explicitly rather than left
|
||||
ambiguous.** R-403 removes a way the route could be DESTROYED between uses; it does not change what
|
||||
the route can be relied on for.
|
||||
- **`07` §8 rows 3b and 4** — unchanged, and that is deliberate: 3b is PROVEN on what the route does,
|
||||
which R-403 does not alter.
|
||||
|
||||
## 11. Teardown — all three layers
|
||||
## 11. Teardown
|
||||
|
||||
- **Machines provisioned:** **none.** No VM, no scratch guest, no drill rig. The drill used the
|
||||
existing guest 9201.
|
||||
- **Hub records created:** **none.** No enrolment, no appliance, no escrow, no claim code.
|
||||
- **On-box artefacts:** the endpoint driver, the password file, the session file and every phase script
|
||||
were shredded or removed; the hollow-unit copy was pulled off as evidence and then deleted; the
|
||||
controller's `settings.json.r102bak` was removed.
|
||||
- **The drilled app:** **docmost is running and healthy with its data back** — HTTP 200, 3 volumes and
|
||||
1 database replayed from its own primary unit, primary and secondary byte-identical again on all five
|
||||
artefacts. All 8 apps on the box report `healthy`. The drill's planted row and accented file remain in
|
||||
the app, as the earlier `felhom_r356b_discriminator` drill left its own.
|
||||
- **Machines provisioned:** none. **Hub records created:** none.
|
||||
- **The drilled app:** `docmost` is running and healthy, and **both copies are complete and
|
||||
byte-identical** — primary and secondary each 3 tars + 4 dumps, 120 082 104 B, sha256 matching. The
|
||||
final Tier-2 run mirrored normally (`114.5 MB, 0 leg(s)`, **0 skips**), which also proves Scenario C1
|
||||
live. The surface shows **no** stale notice on any app.
|
||||
- **On-box artefacts:** the safety-net copy of the unit (deliberately placed OUTSIDE every backup tree,
|
||||
because yesterday's set-aside was swallowed by a directory the product re-created), the endpoint
|
||||
driver, the password and session files and every phase script — all removed or shredded.
|
||||
- All 8 apps on the box report `healthy`.
|
||||
|
||||
## 12. Observations
|
||||
## 12. Register
|
||||
|
||||
1. **FILED: R-403 — after a restore that runs while the primary unit is ABSENT, the next status refresh
|
||||
writes a HOLLOW primary unit.** Measured during the drill: two seconds after the Tier-2 unit restore
|
||||
completed, the 5-minute `backup-cache` job (`internal/backup/backup.go:1116` →
|
||||
`captureAllRecoveryUnits`) rebuilt `backups/primary/docmost/` from a drive with no dump files,
|
||||
producing a manifest carrying `"db_dumps": []` and `"volume_dumps": null`
|
||||
(`evidence-hollow-primary-manifest-1002.json`). The ordinary restore then read it and reported —
|
||||
correctly and uselessly — that the backup held only settings. **The dangerous half is UNMEASURED and
|
||||
is written down as such:** `RunTier2` mirrors the primary unit with `rsyncMirror`, which carries
|
||||
`--delete`, so the next nightly run would plausibly overwrite the good secondary copy with the hollow
|
||||
one. That is a reading of the code, not a test. Filed with the exact experiment that would settle it.
|
||||
Not fixed here: §12 of the task forbids expanding scope, and this is a capture-path change.
|
||||
| Row | Action |
|
||||
|---|---|
|
||||
| **R-403** | **CLOSED** — controller v0.230.0, proven live both ways. Compressed into `CLOSED-ITEMS.md` naming `git show 66156c619fd2:…/OPEN-ITEMS.md` for the original |
|
||||
| **R-404** | **FILED and deliberately NOT acted on** — a decision for Viktor on whether a documents-only push should be subject to the golden-currency gate. Both sides stated, plus what happens if he does nothing. **The gate was not changed.** |
|
||||
| **R-242** | appended — **seventh conviction**, and the first where the day-0 ground does NOT apply |
|
||||
|
||||
2. **FILED: R-242 — the golden-currency gate convicted for the sixth time; the debt was then PAID the
|
||||
same day.** The first `felhom.eu` push used `git push --no-verify` (a **bypass, not a waiver** — the
|
||||
gate offers a waiver only for a release that deliberately needs no golden, and this one needed one),
|
||||
with the day-0 ground re-checked rather than reused. **The operator then instructed the bake, and it
|
||||
was done:** golden 0.229.0 baked, published, round-trip verified, vouched and the floor raised; the
|
||||
gate went **red → green** and the declared bypass is now historical. **The half of R-242 that stays
|
||||
open is untouched: nothing gates the VOUCH itself**, so a baked-but-unvouched golden still passes the
|
||||
currency gate silently. Cadence, measured not asserted: **two bakes in one day** (0.228.0, 0.229.0).
|
||||
Register size: `OPEN-ITEMS.md` **594 → 593** lines (R-403 out, R-404 in); `CLOSED-ITEMS.md` **239 →
|
||||
240**; `ROADMAP.md` unchanged (it carries no R-403 row; `one_register_gate.py` green).
|
||||
|
||||
3. **NOT-A-FINDING: it is my own process error, and its durable home is the MEMORY, not the product
|
||||
register — which is where the first two instances already live and where I have added this one
|
||||
(`credentials-file-values-are-quoted`). A register row would put an operator-facing product
|
||||
backlog entry on a mistake in how I read a credentials file. THIRD INSTANCE, and it changed the
|
||||
box, so it is written down in full.** I read
|
||||
`POST /login` returning 200-with-the-login-page as *"the shared demo password has drifted again"* and
|
||||
**changed the box**: I re-set `password_hash` in the controller's `data/settings.json`. **The password
|
||||
had not drifted.** Values in `~/.config/credentials` are **single-quoted**; my extraction stripped only
|
||||
`"`, so I sent a 15-character string where the password is 13. That is exactly what the memory
|
||||
`credentials-file-values-are-quoted` records, and exactly what the **v0.228.0** report recorded on this
|
||||
same box **on this same day**. Repaired: the hash was re-set to `bcrypt(<correctly unquoted PASSWORD>)`
|
||||
and login verified (302 + `felhom_session`), so the end state matches what the v0.228.0 session
|
||||
independently verified. **What I cannot claim:** that the original hash bytes were restored — I deleted
|
||||
my own `settings.json.r102bak` before finding the error. The end state is correct **by verification,
|
||||
not by restoration**, and the drill README says so.
|
||||
## 13. Observations
|
||||
|
||||
4. **NOT-A-FINDING: the local unit restore's volume leg now goes through the R-354 `volumeReplayFrom`
|
||||
seam.** Strictly this is a line the task did not list. It is not a new seam — it is the one the
|
||||
off-site path already uses, it defaults to the real `restoreDockerVolumesFrom` so production is
|
||||
byte-identical, and without it the acceptance test could assert only that the restore succeeded, not
|
||||
which directory the tars came out of. §10 of the task requires asserting the source directory, and for
|
||||
40 of 53 apps that archive is the whole dataset.
|
||||
1. **FILED: R-404 — six correct bypasses of one gate is a habit, not a guard.** Filed as a decision,
|
||||
not built. Detailed above.
|
||||
|
||||
5. **NOT-A-FINDING: `fmtTimeStr` was extracted to a package-level `fmtRFC3339Local`.** The confirm (a
|
||||
template) and the outcome (Go) both name the copy's date. Two renderings that could disagree is how a
|
||||
customer confirms one date and is told another. One implementation, two callers.
|
||||
2. **FILED: R-242 — the seventh conviction, and the ground that justified the other six has expired.**
|
||||
The `felhom.eu` push used `git push --no-verify`, declared. Unlike the previous six, **this release
|
||||
does bite a day-0 box**: R-403 is a defect in the nightly Tier-2 copy, which a new machine starts
|
||||
running on its first night.
|
||||
|
||||
6. **NOT-A-FINDING: an import-root bind would be mirrored under `hdd/` by Tier-2.** `tier2DestRel` maps
|
||||
everything that is not `RootUserdata` to `hdd`, including `RootImport`. No catalogue template
|
||||
classifies an import bind as anything but `excluded`, so nothing reaches it today and the count in §9
|
||||
is unaffected. Noted while reading `tier2_capture.go` for Part 3; not acted on, and not filed, because
|
||||
it is unreachable from the current catalogue.
|
||||
3. **NOT-A-FINDING: my own live validation found a defect my unit tests did not, and the shape is
|
||||
worth naming.** The first draft flagged "the package is older than the run" by comparing dates —
|
||||
true of **every healthy app**, because a unit is always captured shortly before the run that
|
||||
mirrors it. Four healthy apps on the box would have been warned. It is not a register row because
|
||||
it was found and fixed inside this task, but it is recorded in `CONTEXT.md` as a shape: **a warning
|
||||
that fires on everything costs the same as the comforting lie it replaces.**
|
||||
|
||||
7. **NOT-A-FINDING: `07` §6.2's citation `felhom.eu/REPORT.md:17-24` is dead.** `REPORT.md` is
|
||||
overwritten every session by convention, so the Phase-0 count's source no longer exists — which is why
|
||||
§6.2 said the two methods could not be diffed from the repo. The corrected §6.2 marks the citation as
|
||||
since-overwritten rather than silently dropping it.
|
||||
4. **NOT-A-FINDING: my first Scenario-D control was broken and produced a false alarm.** I scanned a
|
||||
fixed 9000-character window from each app's name, which spilled into the next app's row, so kimai
|
||||
appeared to carry docmost's warning. Re-done by splitting on the real row container. **The broken
|
||||
control is what surfaced observation 3**, so it is recorded rather than quietly replaced.
|
||||
|
||||
## 13. Housekeeping — register size
|
||||
5. **NOT-A-FINDING: `rsync` is not installed in guest 9201** — it lives inside the controller
|
||||
container. My first repair script shelled out to it with `set -uo pipefail` (no `-e`) and silently
|
||||
did nothing; the log says so at the top of `phase1d-repair.log`. Same trap as yesterday's
|
||||
`docker volume rm` in a different disguise: **an unchecked exit code that looks like success.**
|
||||
|
||||
| File | Before | After |
|
||||
|---|---|---|
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | 596 lines | **593** — 4 closed rows moved out (R-102, R-103 and their C9-F4 / C9-F1b aliases), 1 new row (R-403) |
|
||||
| `documentation/backlog/CLOSED-ITEMS.md` | 237 lines | **239** — 2 compressed entries, each naming `git show 1623a4d5b5d5:documentation/backlog/OPEN-ITEMS.md` for the full original |
|
||||
| `documentation/backlog/ROADMAP.md` | 160 lines | 160 — carries no R-102/R-103 row; `one_register_gate.py` green |
|
||||
6. **NOT-A-FINDING: one `--no-verify` was used unnecessarily.** The evidence/script push at
|
||||
`66156c6` was pushed with `--no-verify` before I had checked whether the gate was green — it was
|
||||
(the immediately following no-op push printed `gates OK`). Harmless, and recorded because a bypass
|
||||
that was not needed is exactly the habit R-404 is about.
|
||||
|
||||
7. **NOT-A-FINDING: `recordTier2Success` and `tier2UnitConfirmMsg` kept their old signatures as thin
|
||||
callers.** Both needed new arguments, and both had existing callers including tests. §9/E2 forbids
|
||||
editing an existing test, so each gained a `…WithUnit` / `…WithStaleness` core with the old form as
|
||||
the thin caller — the ONE-implementation-two-callers pattern this repo already uses. No existing
|
||||
test was touched.
|
||||
|
||||
8. **NOT-A-FINDING: the drill's session expired mid-run and a POST silently did nothing.** After the
|
||||
0.230.0 restart the recorded `felhom_session` was dead; `ctl.sh post` printed no status line and no
|
||||
Tier-2 ran. Caught because the secondary was unchanged when it should have been evaluated. Recorded
|
||||
in the evidence as `phase3-scenarioB-first-attempt-session-expired.log` rather than deleted.
|
||||
|
||||
## 14. Final verification
|
||||
|
||||
```
|
||||
felhom-controller/controller$ go build ./... && go vet ./... && go test ./... → all ok
|
||||
felhom-controller$ python3 controller/scripts/controller_gates.py → all 13 gates OK
|
||||
felhom.eu$ python3 scripts/repo_gates.py → all 12 gates OK
|
||||
felhom.eu$ python3 scripts/test_read_credential.py → OK
|
||||
felhom.eu$ python3 scripts/repo_gates.py → 11 OK, golden-currency FAILED (declared, §13.2)
|
||||
```
|
||||
|
||||
## 15. Delivery — the bake and the vouch (added after the operator instructed it)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `GOLDEN_VERSION` | **0.229.0** |
|
||||
| `GOLDEN_SHA256` | `39aa886df77b21757aef3b298a389343dc0df5134bb0f14e8f92a451d7bdae87` |
|
||||
| size | **656 864 331 B** |
|
||||
| round trip | downloaded bytes match the bake on **both** size and sha |
|
||||
| delivered artifact | `./etc/felhom-controller-image` read out of the DOWNLOADED archive → `felhom-controller:0.229.0` |
|
||||
| third reader | the hub's own Day-0 dropdown read the same sha from Gitea, before anything was vouched |
|
||||
| acceptance markers | 1/1/1/1 present, 0/0/0 absent, counted on the committed log |
|
||||
| 404 pre-gate | proven with a positive control (0.228.0 → 200) before the 404 was believed |
|
||||
| token leak grep | proven with a positive control (a seeded copy grepped 1) before the 0 was believed |
|
||||
| vouch | three fields, verified by **re-reading the manifest** — not the flash |
|
||||
| R-120 gate | **passed**, not bypassed (fleet newest 0.229.0, golden 0.229.0) |
|
||||
| floor | raised to 0.229.0; **proven ACTING** — `demo-felhom` self-updated `0.228.0 → 0.229.0` unattended |
|
||||
| teardown | build guest 9100 `--purge`'d, token/script/log shredded **after** the log was copied out, VM powered off, disk reverted to `virgin` |
|
||||
|
||||
Full record: `felhom.eu/documentation/tests/golden-0.229.0-2026-08-31/`.
|
||||
|
||||
@@ -1220,6 +1220,37 @@ DIRECTORY, so the same restore that always worked from the primary now works fro
|
||||
of 1, 28.65 s, an accented filename byte-identical, and `secrets recovered=2/2` with the guest's
|
||||
`app.yaml` also moved aside:
|
||||
`felhom.eu/documentation/audits/DRILL-r102-tier2-unit-2026-08-31/`.
|
||||
- **Since v0.230.0 it also REFILLS the app's own drive** before returning — see the R-403 note below.
|
||||
|
||||
**The nightly copy refuses to replace a complete package with an empty one (R-403, v0.230.0)** —
|
||||
`internal/backup/r403_hollow.go` + the precondition in `RunTier2`.
|
||||
|
||||
**The measurement, because this was run before it was fixed.** On the shipped v0.229.0, on `demo-hp`:
|
||||
an app's Tier-2 copy went from **120 082 104 B (4 database dumps + 3 volume tars) to 7 036 B (none of
|
||||
either) in one nightly run**, reported as a success. `RunTier2` guarded the unit leg with `os.Stat`
|
||||
alone, `rsyncMirror` is `rsync -a --delete`, and nothing compared the two sides — and an empty
|
||||
recovery unit is a folder that exists. Evidence:
|
||||
`felhom.eu/documentation/audits/DRILL-r403-tier2-delete-2026-08-31/`.
|
||||
|
||||
- **The predicate asks the MANIFEST, never the byte size.** `unitCarriesData` is true when the unit's
|
||||
manifest lists a database dump or a volume tar. Absent or unparseable manifest ⇒ hollow, fail closed.
|
||||
- **The refusal is one shape only:** source hollow AND destination not. complete→complete,
|
||||
complete→hollow and hollow→hollow all mirror as before. **`--delete` stays and the data legs are
|
||||
untouched** — §8 row 5's derived-copy rebuild is a design decision and a copy that legitimately
|
||||
shrinks still shrinks.
|
||||
- **The other legs still run** and the run is not failed; a preserved package must not cost the
|
||||
customer their file legs or raise a red alarm on a healthy box.
|
||||
- **What the surfaces say.** A preserved package is older than the run that preserved it, so the
|
||||
per-app card carries „A másolat adatcsomagja régebbi, mint a legutóbbi mentés…", and the
|
||||
„Teljes visszaállítás a másolatból" confirm names the **package's own date** (from the mirrored
|
||||
manifest's `created_at`) plus a `FIGYELEM` clause saying why. The restore **outcome** names the same
|
||||
date. Only an app whose leg was actually preserved shows any of it — a warning that fires on
|
||||
everything costs the same as the comforting lie it replaces.
|
||||
- **The cause is closed too:** `RestoreTier2Unit` refills an absent or hollow primary unit from the
|
||||
mirror **inside the call**, because the hollow manifest was written two seconds later by the
|
||||
5-minute capture job. Never over a complete primary, never after a failed restore, and **the capture
|
||||
itself is not guarded** — it describes reality, and with the primary refilled there is nothing hollow
|
||||
left to describe.
|
||||
|
||||
**Per-app Tier-2 config panel (v0.57.0)** — `GET/POST /stacks/{name}/backup`
|
||||
(`internal/web/tier2_config_handler.go` + `templates/tier2_config.html`). The "2. mentés" row's
|
||||
|
||||
Reference in New Issue
Block a user