admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Correct the placement mis-framing, and file what we wrote down and never filed (R-368..R-375)
gates / gates (push) Successful in 16s
Details

Browse Source 
Documentation and survey only. No code, no machine contacted.

THE CORRECTION. The 40 catalogue templates without a configurable path are not missing a
choice: 01-topology-and-trust.md:150-152 classes each volume hot (DB/config/cache -> fast
storage, ENFORCED) or bulk (media/files), and the 40 are all-hot apps. The deploy page has
been saying so to the customer all along (deploy.html:624-625). SPEC-app-data-placement and
R-352 are corrected in place with the framing MARKED, not deleted; every measurement stands.
R-356 was re-checked and survives, strengthened - an absent HDD_PATH is the normal state, so
reading it as "not installed" misreads a correct configuration.

The disk claim, precisely: since R-165 there is ONE guest data volume with two binds, not two
volumes (build-golden.sh:29-40, 99). A physical-disk failure losing data and first-tier copy
together is REAL and is what the other tiers exist for. A full data volume stopping the OS is
NOT real and was the overstated one.

THE SWEEP. 113 survey-class documents examined, 14 statements of "not filed", 2 already filed.
Its positive control convicted the sweep itself twice before it convicted the corpus - markdown
bold broke the strongest pattern, and the reporter re-searched a truncated line - both false
zeros of the exact class being hunted, and together worth 2 of the 14.

THE HEADLINE. The gap the 2026-08-21 drill rediscovered WAS filed - as R-107, ROADMAP.md:122,
M/READY, 2026-07-28 - and is absent from OPEN-ITEMS.md, which calls itself the single source of
truth. OPEN-ITEMS and that rule both landed 2026-07-27; R-107 went to ROADMAP alone the day
after. 72 ids live only in ROADMAP, 29 not done, some of them findings. Filed as R-369 (HIGH).

Five more still-open gaps filed with their ages: R-371 (17d), R-372 (38d, the oldest), R-373
(20d), R-374 (14d), R-375 (4d). R-368 corrects Part 4: the storage default IS applied at deploy
time via deploy.html:612 - the earlier "the deploy route never reads it" came from grepping Go
and never the templates. R-370 records the process failure and is closed by the template change.

PROMPT-TEMPLATE gains the two rules it lacked: name the architecture document for the area and
say what it says (with a file->area map and the test "is this something we chose?"), and an
enumerated gap becomes a register row in the same session - a ROADMAP row alone does not count.

Ceiling R-367 -> R-375.
This commit is contained in:
authentik Default Admin
2026-08-22 11:18:08 +02:00
parent 5a7502b6d5
commit 091a4b7444
6 changed files with 822 additions and 346 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+266 -342
View File
@@ -1,383 +1,307 @@
# REPORT — the database nobody backed up, and the restore that returned most apps nothing (2026-08-22)
# REPORT — correcting what we mis-called a defect, and finding what we wrote down and never filed (2026-08-22)
**Controller v0.217.0 → v0.218.0.** Two fixes, both found by watching a machine on the night of
2026-08-21, both confirmed the same way. **Part 1 first, because it is the only place in the product
where one customer action causes permanent total loss.**
The drill that found them is preserved at
`documentation/audits/REPORT-DRILL-backup-truth-2026-08-21.md`; its evidence is in
`documentation/audits/DRILL-backup-truth-2026-08-21/evidence/`.
**Documentation and survey only. No code, no version bump, no bake, no deploy, no machine contacted.**
The previous report is preserved at `documentation/audits/REPORT-v0.218.0-r354-r355-2026-08-22.md`.
---
## 0. Baselines, re-established — not trusted from the sheet
## 1. §2's four claims — ALL FOUR HELD. No halt.
| item | expected | confirmed |
| # | claim | verdict |
|---|---|---|
| controller | 0.217.0 → 0.218.0 | ✔ repo head `v0.217.0`; live on `demo-hp` `…:0.217.0` |
| agent | 0.130.0 | ✔ repo head and `felhom-agent --version` on the box |
| golden vouched | 0.217.0 | ✔ `<option value="0.217.0" … selected>` |
| floor | 0.217.0 | ✔ „Effective floor: v0.217.0 — source: DB (hub_settings)" |
| min agent | 0.129.0 | ✔ |
| register ceiling | R-366 | ✔ (now **R-367**) |
| `ssh hp` → 192.168.0.104 | works | ✔ |
| clean trees, all four repos | `HEAD == origin/main`, 0 dirty | ✔ |
| 1 | `07-backup-architecture.md:297` — 53 templates, 52 named volumes, exactly 13 with a `backup:` block, and those 13 are exactly the ones that bind a configurable path | **HOLDS**, at `:296-299`. The heading above it reads *"Coverage per app class — and an unresolved count"*, which the sweep then followed |
| 2 | `_recovery-inventory-2026-07-28.md` Tier-3 row — *"…unpacked by no offsite action… no single action does it and no UI routes it. This is my own enumeration; it is not currently filed as a finding."* | **HOLDS**, verbatim, at `:373` |
| 3 | `:392-400` — `rootfs`, `mp0 /var/lib/docker`, `mp1 /mnt/sys_drive` as separate volumes | **HOLDS**, at `:392-396` — **but it describes a layout no current box has.** See below |
| 4 | `00-capability-map.md:94` — one data volume; a local backup bounded by free space | **HOLDS**, at `:94` |
**Claims 3 and 4 describe different layouts, six days apart, and 4 is the live one.** The inventory
(2026-07-28) records the pre-R-165 split. `felhom-agent/configs/build-golden.sh:29-40` — live source —
bakes **ONE** data volume at `/var/lib/felhom`, with `/var/lib/docker` and `/mnt/sys_drive` as two
**binds of that same volume**, and states at `:99` that *"There is deliberately no mp1"*.
**So the prompt's own §1.2 instruction is out of date:** it asked me to say that named volumes and
first-tier backups are *"on two different guest volumes on one physical disk"*. They are on **one**
guest volume, by two binds. The disk claim is stated precisely in §4 below.
**A correction to my own §2 working.** In verifying claim 2 I reported that a literal grep matched
`:373`. It did not — it returned nothing, and what printed was the second grep in the same cell.
`**not** currently filed` does not match `not currently filed`, because of the markdown bold. The
claim still holds (I read the line), but the check I quoted was wrong — and that false zero turned out
to matter, see §3.
---
## 1. PART 1 — where the wrong name comes from
## 2. The sweep — three counts, and the control that earned them
**It comes from a guess, made in a place where an answer was available.**
**Method** (`scripts` not committed; it is a throwaway, and the terms below are the durable part):
enumerate every **survey-class** document — inventory / audit / spike / review / diagnosis / recon /
campaign / report / validation / rehearsal / walk / finding / spec / triage — under
`documentation/{audits,architecture,pilot,backlog}`, excluding runbooks, templates, READMEs,
CHANGELOGs and ROADMAP, because those instruct rather than conclude. For each, ask whether any
`OPEN-ITEMS.md` row cites it by filename, then search every line for the **shapes** an unfiled gap
takes.
`deriveStackName` (`controller/internal/appbackup/dbdump.go:770-798`) is handed a container name and
asked which app owns it. For `paperless-postgres` it strips the role suffix to `paperless`; finds
`paperless` is **not** a deployed stack; finds the container name is not one either; finds no known
stack is a prefix of it (`paperless-ngx` is not a prefix of `paperless-postgres`) — **and then returns
the unresolved candidate anyway**, on its last line, silently.
| count | value |
|---|---|
| **documents examined** | **113** (survey-class; 114 after this session added one) |
| **gaps found** (documents saying, in their own words, that something was not filed) | **14** statements |
| **of those, already filed** | **2** — and the other 12 break down in §3 |
**Every place that value is used, and what each did with it:**
**The search terms, so the next person can widen them:**
| consumer | file:line | effect |
```
not (currently )?filed never filed no finding not a finding
unresolved unfiled disagreement no single action
nothing (does|routes|reads|consults) never been should be
ought to is not (currently )?(tested|proven|observed|covered|wired)
has (never|not) been (seen|observed|walked|proven|done)
no (test|row|register) (pins|covers|cites) TODO FIXME ⚠ not covered gap
```
Every word-gap in the first four patterns is `[*_`~ ]*`, i.e. **markdown emphasis is tolerated** —
see §3.
### The positive control — and it convicted the sweep twice before it convicted the corpus
**Plant → find → remove → fail to find**, on a scratch copy of the whole `documentation/` tree:
| step | strongest-shape hits |
|---|---|
| unplanted scratch copy | **14** |
| a synthetic gap planted, **bolded and on one line** | **15** — convicted by name and quoted back |
| scratch discarded, real corpus re-run | **14** |
**The control found two defects in my own sweep before it found anything in the corpus, and both were
false zeros of exactly the class this session is about:**
1. **Markdown emphasis broke the strongest pattern.** The single most important line in the corpus is
written `it is **not** currently filed as a finding`, and `not (currently )?filed` does not match
it. Fixed by tolerating `[*_`~ ]*` between words.
2. **The reporter re-searched a truncated copy of the line.** Hits were stored as `line[:200]`; that
line is a long table row and the phrase sits past column 200, so the detector caught it and the
**report** dropped it. Fixed by matching the full line and truncating only for display. This alone
moved the count **12 → 14**.
A sweep whose first two runs would have under-reported by 2 is exactly why the control is mandatory.
---
## 3. What the 14 were, judged
| verdict | n | which |
|---|---|---|
| dump directory | `backup/backup.go:482,506` | wrote to `…/primary/**paperless**/db-dumps/` on the SYSTEM drive |
| dump filename | `appbackup/dbdump.go:200-212` | `paperless-postgres.sql`, not `paperless-ngx-postgres.sql` |
| unit assembler | `backup/recovery_unit.go:131` | reads `…/primary/**paperless-ngx**/db-dumps` → finds nothing → `db_dumps: null` |
| off-site collector | `backup/offbox_capture.go:32` | resolves from the unit path → the orphan is invisible to it |
| restore's DB leg | `backup/restore_db.go:69` | `db.StackName != stackName` → never replays |
| **safety-dump filter** | `backup/offbox_reconstitute.go:134` | `mine` empty → `hasDB=false` → **no undo copy, and the refusal is never reached** |
| `.fab` export | `appexport/export.go:600` | same filter — the bundle carries no database either |
| the "has a DB" flag | `web/handlers.go:1245` | the app displays as having no database |
| **never a gap** — a reasoned "no finding" | 4 | `_design-review.md:9` (nothing deferred), `CAMPAIGN-11:501` (a measurement was honest), and the two **explicit `### Not filed` sections** in `CAMPAIGN-10-two-storage-soak` and `SPIKE-recovery-unit-space` — each item disposed with a reason. **This is good practice, not a failure, and it is what the rest should look like.** |
| **already filed** | 2 | `REPORT-DRILL-backup-truth:321` (its Part 5 became R-360 and R-364, filed the same session) and `REPORT-DRILL:569` (the hub's controller-version blindness — the register already covers it, 2 hits; not re-filed) |
| **still open → NEWLY FILED** | 5 | R-371, R-372, R-373, R-374, R-375 |
| **the headline** | 3 | `_recovery-inventory:373`, `:1043` and `07-backup-architecture.md:337` — all three the same gap, and it turns out it **was** given a number |
**Corroboration from the code itself:** `ListDumpFiles` (`dbdump.go:562`) parses a dump filename under
the comment *"Parse stack name and DB type from filename: `paperless-ngx-postgres.sql`"* — the reader
was written for a name the writer never produced.
### The headline: it was filed. In the other register.
### Controller or catalogue? — **THE CONTROLLER. No halt.**
The gap the 2026-08-21 drill rediscovered **was already numbered R-107**, with a full write-up:
Every container the controller starts already carries `com.docker.compose.project`, read live:
> `ROADMAP.md:122` — *"**No offsite action unpacks the named-volume tars Tier-3 captures on every
> run.** `ReconstituteFromOffsite` skips the unit outright … `PlaceOffsiteRestore` places it only when
> the live unit is ABSENT"* — **M, READY, 2026-07-28**
and cross-referenced twice in `07-backup-architecture.md` (`:337`, `:902`). **It is absent from
`OPEN-ITEMS.md`**, which opens with the words *"the single source of truth for open work"*.
**The dating makes it a rule violation, not a gap in the rules.** `OPEN-ITEMS.md` was created and the
template gained its *"single source of truth"* bullet on **2026-07-27** (`655b69f`). R-107 went into
ROADMAP alone on **2026-07-28** (`070b0ce`) — **the day after**.
**Measured scope: 72 `R-` ids are in ROADMAP and not in OPEN-ITEMS; 29 are not marked shipped, closed
or killed.** Most of the 29 are feature *ideas*, which arguably belong only in ROADMAP. A minority are
**findings**: R-30, R-31, R-32 (all P2-HIGH), R-35, R-40, R-76, R-79, R-25, R-49, R-10, R-107.
**The risk is not double-minting** — the two files share ids and OPEN-ITEMS' ceiling (368) is above
ROADMAP's (331). **The risk is rediscovery**: a session greps one file, finds nothing, and redoes the
work. That is precisely what happened, and it cost an evening and a night. **Filed as R-369, HIGH.**
---
## 4. Part 1 — the record corrected
### The specification
`SPEC-app-data-placement-2026-08-21.md` now opens with a marked **⚠ CORRECTED 2026-08-22** block and
carries four inline `[CORRECTED 2026-08-22]` marks. **Nothing was deleted; every measurement stands.**
What it got wrong: it treated the absence of a storage field on 40 templates as a choice being denied.
The architecture states the rule and states it as **enforced**:
> *"**App data placement is per-volume, not per-app:** `.felhom.yml` classifies each volume **hot**
> (DB/config/cache → fast storage, **enforced**) vs **bulk** (media/files → may be slow)."*
> — `01-topology-and-trust.md:150-152`
The 40 are all-hot apps; the 13 are the bulk apps. **And the deploy page has been telling the customer
exactly this the whole time** — `deploy.html:624-625`: *„A kiválasztott meghajtón az alkalmazás
**fájljai** (média, dokumentumok) tárolódnak. Az **adatbázis a gyors belső SSD-n** fut."*
### The disk claim, stated precisely
- **REAL:** a **physical-disk failure** loses the app's data and its first-tier copy together. That is
what the off-site and whole-machine tiers exist for, and it is **equally true of a drive-resident
app**, whose unit sits beside its data on the drive deliberately so a restore needs the drive and
nothing else.
- **OVERSTATED:** a **full data volume stopping the operating system.** The OS rootfs is a separate
volume, and the capture floor refuses per app before exhaustion (`00-capability-map.md:94`).
Watched working 2026-08-21 with `/var/lib/felhom` at 99%: the reserve refused one app per run, told
the hub, and all 15 containers stayed healthy.
- **WITHDRAWN:** the comparison to Tier 2's same-disk refusal (`tier2.go:329`). Tier 2 refuses a
**second** copy on the same disk; Tier 1's unit is *meant* to sit beside the data.
### Rows re-framed
- **R-352** — measurements (1)–(4) all stand; the conclusions drawn from (1) and (3) are marked
re-framed. (2) is now filed on its own as **R-368**; (4) needs no ruling.
- **R-356** — **re-checked and it SURVIVES UNCHANGED, strengthened.** Because the 40-class correctly
has no `HDD_PATH`, a restore that reads that as *"the app is not installed"* is misreading a correct
configuration. One sentence in it that leaned on the old framing is corrected in place.
### My own errors, named — R-370
**Four instances, not three.** The prompt said three; the evidenced count is four, all authored
2026-08-21: SPEC §1, SPEC §2.3, SPEC §5, and R-352 point (3). Recorded as a **process** failure with
its mechanism — the register and live source were read, `documentation/architecture/` was not — and
the missing step is now in the template. Also written into `CONTEXT.md`.
---
## 5. What still needs the operator's ruling — **from this specification, almost nothing**
- **The placement question is ANSWERED and the answer is no.** Points 1, 2 and 3 of the spec's §5 are
withdrawn as a live question; hot data belongs where it is.
- **Point 4 is the only thing left**, and it is smaller than it looked — see R-368 below.
- **Point 5 needs no ruling.** The Drives count is honest; it is a wording question the design system
already owns.
**One new thing does want your ruling, and it is R-369:** whether the two registers become one, or
whether a gate enforces that a not-done `ROADMAP` row has an `OPEN-ITEMS` counterpart. Triage the 29
first — most are ideas, a minority are findings.
---
## 6. Part 4 — the three answers, and the finding is smaller and different than believed
**1. Is the default store consulted at deploy time, for the 13 apps that take a path? — YES.**
`internal/web/templates/deploy.html:612`:
```
paperless-postgres paperless-ngx kimai-db kimai romm-db romm
{{else if and .IsDefault (not .NotAllowed)}}selected{{end}}
```
**And that label is the stack name BY CONSTRUCTION, not by luck:** `composeExecCustomEnv`
(`internal/stacks/manager.go:1218-1230`) runs compose with `cmd.Dir` set to
`/opt/docker/stacks/<stack>` and **never passes `-p`**, so compose derives the project from that
directory. Renaming the container in the catalogue would have fixed this one app and left the guessing
for the next one. **No catalogue change was made.**
For a new deploy the default drive **is pre-selected**. `DeployStoragePath` embeds
`settings.StoragePath` (`web/handlers.go:89-99`), so `.IsDefault` resolves.
**So `// new apps use this by default` (`settings.go:453`) is IMPRECISE ABOUT THE MECHANISM, NOT
FALSE.** The earlier claim — *"the deploy route never reads it"* — is **wrong**, and it is wrong for
the same reason as everything else this week: the grep behind it
(`grep -nE 'GetDefaultStoragePath|primaryHDDPath|IsDefault' deploy.go manager.go`) **searched Go files
and never the templates.**
**The residual, and it is the whole finding:** the default lives in the **template**, not the server.
`POST /api/stacks/<n>/deploy` takes `values` verbatim; omit `HDD_PATH` and `withPathVars`
(`stacks/deploy.go:584`) gets `""` and no default applies. **That is why the invariant has no test —
there is nothing server-side to test.** Filed **R-368, LOW**.
**2. What the label promises the customer, quoted:** `storage.html:469` —
**„Legyen alapértelmezett új telepítéseknél"** ("Be the default for new installations"), with the
badge **„Alapértelmezett"** at `:34`. **The promise is kept**: it is the default for new installs,
which is exactly what the pre-selection does. It does not promise that every app's data goes there.
**3. What the Drives count counts:** `countAppsUsingPath` (`web/handlers.go:2162-2175`) counts
deployed apps where `appCfg.Env["HDD_PATH"] == storagePath`. **A named-volume app has no `HDD_PATH`,
so it can never be counted — by construction, not by accident.** The number is honest; it means *"apps
that place bulk data on this drive"*, not *"apps using storage"*. Worth one sentence of wording, not a
ruling.
---
## 2. PART 1.2 — the sweep, proven before its answer was trusted
## 7. Part 3 — the template's two new rules
| step | result |
|---|---|
| scratch copy, unplanted | `MISMATCHES: 1` (rc 1) — the known one |
| **plant a second mismatch** (`kimai-db` → `timetrack-db`) | **`MISMATCHES: 2`**, convicted by name: `stack=kimai container=timetrack-db -> derived=timetrack` |
| plant removed | back to `1` |
| **every mismatch removed** | **`MISMATCHES: 0`, rc 0** — "1" is not a stuck value |
| scratch discarded | `kimai/docker-compose.yml` md5-identical to the real catalogue |
**It had neither.** `enumerat` → 0 hits, `register row` → 0 hits; `architecture` appeared 9 times but
§4 item 4 named only `02-controller-module-map.md`, and the S-1 rule at the end governs *updating* a
design doc, not *reading* one first. **No halt.** There is no separate authoring companion.
**The real count: 1 affected app of 53**, from 15 DB containers checked — `paperless-ngx`.
**Rule 1, in §4 where files are read** — abridged; the full text carries a file→area table for all
eight architecture documents:
> **THE ARCHITECTURE DOCUMENT FOR THE AREA THIS TASK TOUCHES — NAME IT AND SAY WHAT IT SAYS.** Not
> "read the architecture folder": name the file, and state in one line what it rules about this area.
> **A prompt that cannot name one says so explicitly, and that absence is itself recorded** — an
> undocumented architectural decision is how a deliberate design gets "fixed" by someone who did not
> know it was one.
>
> **Three sources, in this order, before any claim: the architecture folder holds the REASONING, the
> register holds the WORK, source holds the TRUTH.**
>
> **And the test that catches it: _is what I am about to call a defect something we chose?_** If it
> was chosen and the choice is wrong, that is **a proposal to change a decision** — it goes to the
> operator as a decision, not filed as a bug. **Cost of learning this (R-370):** between 19 and 22
> August a documented placement decision was called a defect in four places.
**Rule 2, on the `OPEN-ITEMS.md` bullet, where a survey session lands:**
> **AN ENUMERATED GAP BECOMES A ROW, IN THE SAME SESSION. PROSE IS NOT A RECORD.**
>
> This binds **surveys, inventories, spikes, reviews and diagnoses**, not only implementation
> sessions… If a document says a thing is missing, unhandled, unreachable or *"not currently filed"*,
> it does not leave the session as prose. It leaves as a row here, with a rank and an owner. Writing
> *"not filed"* is not a disposition; it is a note that the work was seen and dropped.
>
> **A row in `ROADMAP.md` alone does not satisfy this** … invisible to every standing rule that says
> *"grep the register before minting"* (**R-369**).
>
> **The cost, recorded so the rule can be narrowed later rather than becoming permanent by accident:**
> R-107 … was enumerated on **2026-07-28** … and never entered here. **It was rediscovered from
> scratch 25 days later by an overnight drill**, and shipped as R-354.
Two rules, one place each, ~40 lines added to a 521-line document.
---
## 3. PART 1 — the dumps already written under the wrong name
## 8. Rows opened — ceiling R-367 → **R-375**
**They stay, untouched, and nothing will ever collect them.**
`/mnt/sys_drive/felhom-data/backups/primary/paperless/db-dumps/paperless-postgres.sql`, 312 381 B,
last written 07:38 UTC by the final pre-fix cycle — **verified still present after the fix.**
| id | rank | what | first written down | age |
|---|---|---|---|---|
| **R-368** | LOW | The storage default **does** apply at deploy time (`deploy.html:612`); the earlier "never reads it" was wrong. Residual: the default lives in the template, not the server, so the API has none and nothing server-side can be tested | 2026-08-21 (as the wrong claim) | 1 d |
| **R-369** | **HIGH** | **Two registers.** 72 ids in ROADMAP only, 29 not done, some of them findings. R-107 sat there 25 days and was rediscovered by a drill | 2026-07-28 | **25 d** |
| **R-370** | CLOSED | Process: a documented decision called a defect four times; architecture folder never read. Rule now in the template | 2026-08-19 | 3 d |
| **R-371** | LOW | The off-site tier is the only tier that announces nothing on success | 2026-08-05 | 17 d |
| **R-372** | LOW | A Tier-2 copy that has **never** been produced (source missing) is not surfaced distinctly | **2026-07-15** | **38 d — the oldest** |
| **R-373** | LOW | `SysDataGrowGB` is the intended sizing lever, it works, and nothing sets it | 2026-08-02 | 20 d |
| **R-374** | LOW | Three C1 refusal cases judged borderline, left unfiled **and never named**, so nobody can re-judge them | 2026-08-08 | 14 d |
| **R-375** | LOW | A PBS datastore audit signal noted and explicitly not filed | 2026-08-18 | 4 d |
Nothing deletes them by design: the F5 stale-primary prune (`backup/backup.go:1248`) skips any
directory that is not a deployed app, under the guard *"an undeployed app's last backup is still its
restore point"*.
**They can be adopted, by hand:** move the file to
`…/primary/paperless-ngx/db-dumps/paperless-ngx-postgres.sql` and it becomes a readable restore point.
**Deliberately not automatic** — it would sit beside volume tars from a different time (an incoherent
pair, the shape the R-43/R-44 stamp exists to surface), and a controller that silently relocates a
customer's data on upgrade is a migration, not a fix. **Filed as R-367.**
**Oldest gap recovered: R-372, written down 2026-07-15, 38 days.**
**Most consequential: R-369**, because it explains the other seven.
---
## 4. PART 2 — the mechanism, and the comment corrected
## 9. What was dropped, and observations
`ReconstituteFromOffsite` skipped every placement flagged as the unit (`offbox_reconstitute.go:344`),
and the volume archives live **inside** the unit. A search of the whole off-site path found no
volume-restore call; the only two callers of `restoreDockerVolumes` were `restore.go:67` and
`restore_unit.go:255`, both local.
**Dropped: nothing.** Parts 1, 2, 3 and 4 all completed. Part 4 was droppable-first and was done.
**The comment as it stood:**
**Observations — noticed, not acted on:**
> *"The live recovery unit is still never overwritten — it is the LOCAL restore path's source and
> clobbering it would trade one recovery route for another. The snapshot's dump is replayed from the
> scratch unit instead, so nothing is lost by skipping it."*
**Which half still holds: the FIRST.** The live unit is the local restore path's own source and must
never be clobbered — that is why the skip stays, and scenario D now fingerprints the whole live unit
across the operation to keep it true.
**The second half was false and is corrected, not left.** True of the *database* dump, false of the
*volume* archives, which live in the same unit and were replayed by nothing at all. Skipping the
placement is correct; treating the skip as harmless was not — and that sentence is exactly why a
reader would not look.
**The fix:** `restoreDockerVolumesFrom(stackName, dumpDir) (int, error)` — the local path's own replay
with an explicit directory, the same shape `reimportDBDumpsFrom` already had beside `reimportDBDumps`.
**One implementation, two callers.** Volumes replay **before** the database (a logical dump must still
win over a volume-tar copy of the same database) and **inside the stopped window** (Docker will not
replace a volume a container holds).
- **The two `### Not filed` sections are the model.** `CAMPAIGN-10-two-storage-soak:383` and
`SPIKE-recovery-unit-space:228` list what they decided not to file **and why, item by item**. That
is a disposition, not a shrug. If the new rule needs an exemplar, those are it.
- **`07-backup-architecture.md:337` already names this gap and points at R-107**, so the architecture
doc was right and current while the register was empty. The document was not the weak link.
- **The `_recovery-inventory` cites `07-backup-architecture.md:81` for a phrase that is no longer
there** (`grep 'missing-only merge'` → only the inventory's own copy). A stale line-number citation;
not filed, because the doc it points into has since been rewritten wholesale and the claim it
supported is now handled by R-354.
- **The sweep's "cited by no row" count (89) is not a defect count.** Many audits are cited by the
capability map, by `where-felhom-stands.yaml` or by other reports rather than by a register row. It
is a *candidate* filter, and it earned its keep only in combination with the shape search.
- **`REPORT-hub-blindness.md` exists at the repo root and is cited by no register row**, but the
register does cover its subject (2 hits). Left alone.
---
## 5. THE CUSTOMER'S MESSAGE, BEFORE AND AFTER, VERBATIM
## 10. CI
**R-354 — calibre-web, identical fixture, identical steps, both on `demo-hp`:**
| | |
|---|---|
| **before** (0.217.0, 09:41) | `ok=true` — „A(z) calibre-web: **0 fájl visszaállítva** (mentés: 2026-08-22 09:38) — az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa." …and `ls: /vol/R354-2026-08-22: No such file or directory` |
| **after** (0.218.0, 09:56) | `ok=true` — „A(z) calibre-web: **0 fájl és 1 adatkötet visszaállítva** (mentés: 2026-08-22 09:52) — az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa." …and **5/5 files byte-identical** |
**R-355 — paperless-ngx:**
| | |
|---|---|
| **before** (2026-08-21) | „A(z) paperless-ngx: 0 fájl visszaállítva … **Ennek az alkalmazásnak nincs adatbázisa.**" over a live 72-table PostgreSQL, with **no undo copy taken at all** |
| **after** (0.218.0, 09:57) | „A(z) paperless-ngx: **0 fájl és 3 adatkötet és az adatbázis visszaállítva** (mentés: 2026-08-22 09:52) — az alkalmazás újraindult." |
**A third sentence now exists for the case that had no honest wording** — an app that HAS a database
whose snapshot carried no dump: „… FIGYELEM: ennek az alkalmazásnak **VAN adatbázisa**, de a mentés nem
tartalmazott adatbázis-mentést, ezért az adatbázis **NEM állt vissza**. A visszaállítás előtti állapot
mentése megvan: `<undo>`". That case previously printed the same confident „nincs adatbázisa".
---
## 6. THE HARDWARE WALK
**Method: endpoint-level — the exact endpoints the dashboard's JS calls (`/api/backup/run`,
`/backup/offbox/run`, `/backup/offbox/restore`, `/backup/offbox/reconstitute`), with no shell inside
the machine for any step of the walk.** Planting the fixture and reading the result back used a shell
and is setup/verification, not the walk. No browser exists on DooPlex.
**The fixture, hashed before anything ran:**
```
7708bf6582f990ee5c98e3fb6638214d4916b517ea80638baf20b0662b56506e SENTINEL.txt
fb67d42bdfe09815922c2f4f4a2086025bd1a074757d17b5e5164d29b1a5e8d5 binary-512k.bin
847fbae0868fc9a2c985d7411fa545c4853e8561f7b8cc8dd485437b1f1d76c4 nested/őszibarack.md
f15b6d0f88e5499836f585403e3b59b4f2006a0e6a96ce4ec598dd7abe38bff5 plain.txt
857e8594005d11f9802f61018603b7746e7af37ac02631aaef2db57f9700fb58 árvíztűrő-tükörfúrógép.txt
accented names as RAW BYTES (UTF-8 NFC):
árvíztűrő-tükörfúrógép.txt = c3a1 72 76 c3ad 7a 74 c5b1 72 c591 2d 74 c3bc 6b c3b6 72 66 c3ba 72 c3b3 67 c3a9 70 2e 74 78 74
őszibarack.md = c591 73 7a 69 62 61 72 61 63 6b 2e 6d 64
```
**The comparator was proved able to convict first:** one byte flipped at offset 300 000 of
`binary-512k.bin` (`3b` → `00`) → `binary-512k.bin: FAILED`, rc 1, the other four `OK`; the unmodified
set rc 0. Mutant discarded.
### Results
| scenario | result |
|---|---|
| **1-A** the dump is inside the unit, and inside the off-site snapshot | **PASS.** 0.217.0 09:37: `db_dumps = None`, dump refreshed into the phantom dir. 0.218.0 09:45: `db_dumps = ['paperless-ngx-postgres.sql']` (312 669 B) in the app's own unit, and `restic ls` shows it inside the snapshot **for the first time** |
| **1-B** an undo copy taken and verified before anything stops; the message names the database | **PASS.** Before: `find /mnt -name "pre-restore-*paperless*"` → **0**. After: `pre-restore-20260822T075658Z-paperless-ngx-postgres.sql`, 312 957 B, in the app's own unit |
| **1-C** the undo cannot be taken → the whole restore refuses, nothing changed | **PASS**, on the app that could never reach this guard before. `ok=false`; the marker kept its mutation (`51d37b0f…`) and `StartedAt` was unchanged — **the app was never stopped** |
| **1-D** every other app byte-identical | **PASS.** `romm-mariadb.sql`, `kimai-mariadb.sql` unchanged in name and location; the only phantom directory is the one pre-existing orphan; no new one created |
| **2-A/B** the volume comes back and the message says so | **PASS.** calibre-web 5/5 byte-identical incl. both accented names; paperless-ngx 3 volumes + the database |
| **2-C** a snapshot with no volume archives is unchanged | **PASS** — the unit test asserts the byte-identical sentence; live, romm/kimai unaffected |
| **2-D** the live recovery unit is never written | **PASS** — fingerprinted across the whole operation (see red-proof 6) |
| **2-E** a failed replay is reported as a failure naming the volume | **PASS** (unit test; the live path returns the same error) |
All 15 containers healthy after the walk.
---
## 7. RED-PROOFS — seven, each mutation asserted applied and reverted
| # | mutation | outcome |
|---|---|---|
| 1 | `resolveStackName` ignores the compose label | **FAIL as required.** `= "paperless", want "paperless-ngx"`, and the consequence printed both paths — written to `…/primary/paperless/db-dumps` vs read `…/primary/paperless-ngx/db-dumps`. **The empty database record, returned** |
| 2 | outcome message back to the counter-only predicate | **FAIL as required**, reproducing the 21 August sentence verbatim: `"A(z) paperless-ngx: 0 fájl visszaállítva — az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa."` |
| 3 | the fail-closed refusal removed from `writeSafetyDump` | **FAIL as required** — *a restore was seen proceeding with no undo copy* |
| 4 | the volume leg removed | **FAIL as required.** `VolumesReplayed = 0` and the replay never called — last night's silent loss, returned |
| 5 | the volume count dropped from the message | **FAIL as required**, reproducing „A(z) calibre-web: 5 fájl visszaállítva …" verbatim |
| 6 | the live-unit guard removed | **PASSED FIRST — a defect in MY TEST, not the code.** The fingerprint had been narrowed to the volume directory and was blind to a placement writing into the unit **root**. Widened to the whole unit (excluding only the documented undo copies) it convicts: `PLACED:17603ba1…` appears. **The R-181 class, reproduced inside its own regression test — and the reason the red-proof is mandatory** |
| 7 | the volume-replay error swallowed | **FAIL as required** — a partial replay reporting success |
**Green gate:** `go build ./... && go vet ./... && go test ./...` — **28 packages ok, 0 FAIL lines.**
Controller gates: all 11 OK. felhom.eu gates: all OK except the golden-currency gate, which was red
until the bake (§8) — correctly, and never bypassed.
*(Instrument note: `go test ./... | grep -vE '^ok'; echo rc=$?` reports the **grep's** exit code, which
is 1 when every test passed and nothing was left to print. The verdict above is from an explicit
`FAIL`-line count, not from that.)*
---
## 8. WHAT THIS DOES **NOT** FIX — the blocker for the apps that need it most
**The off-site restore still refuses outright for the 40 of 53 apps that declare no data drive
(R-356)**, telling the customer that a running app „nincs telepítve" and to reinstall it "to the same
place" — which those apps give them no way to choose. Those are exactly the apps whose entire dataset
is a named volume, **so R-354's fix cannot reach them until R-356 is closed.**
That is why the live confirmation used `calibre-web` and `paperless-ngx`: they declare a drive and can
actually reach the restore. R-356 is out of scope by the task's own list, and it is now the first thing
worth doing.
---
## 9. REGISTER
- **R-354 — CLOSED**, shipped + proven-live (v0.218.0).
- **R-355 — CLOSED**, shipped + proven-live (v0.218.0).
- **R-367 — OPENED (LOW)**: dumps already written under the wrong name are stranded; adoptable by hand,
deliberately not automatic, never on an upgrade path.
- **Ceiling moved: R-366 → R-367.**
---
## 10. DELIBERATELY OUT OF SCOPE — so it does not read as forgotten
R-353 (the empty restore reported as success — real, and next), R-352 (where the 40 apps' data lives,
awaiting a ruling), R-356 (§8), deploy-and-restore as one act, R-360 (the delete guard on verification
copies), R-364 (the accented-search instrument defect), and the remaining drill rows R-357–R-359,
R-361–R-363, R-365, R-366.
---
## 11. COMMITS, VERSIONS, CI
| | |
|---|---|
| `felhom-controller` | `5ce3a44` the fix + tests; `2da259a` README + CONTEXT |
| image | `gitea.dooplex.hu/admin/felhom-controller:0.218.0` (150M) |
| deployed | `demo-hp` guest 9201 — `…:0.218.0 Up (healthy)` |
| `demo-felhom` | **untouched this session**, still 0.217.0 |
---
## 12. BAKE
`RUNBOOK-manual-build.md` §4.0/§4.1, in the drill VM. **The golden-currency gate went red the moment
v0.218.0 was released and stayed red until the bake — that is correct, and it was never bypassed.**
| | |
|---|---|
| `GOLDEN_VERSION` | **0.218.0** |
| `GOLDEN_SHA256` | **8e427869d13eafb71562b77d1535eef6c7f32b4db24f659b988ec6d7db8f478b** |
| package | `…/generic/felhom-golden/0.218.0/golden.tar.zst`, 657 026 013 B |
| template | `debian-13-standard_13.6-1_amd64.tar.zst` — listed fresh, not assumed |
**Pass markers, each checked, with the negative controls:**
```
docker OK (overlay2 : 1 -> " docker OK (overlay2; data-root /var/lib/docker)"
including mount point : 2 -> rootfs ('/') and mp0 ('/var/lib/felhom') [there is no mp1]
upload OK (HTTP 201) : 1 -> pre-delete HTTP 404 (404/204 expected)
excluding : 0 <- negative control
FATAL : 0 <- negative control
```
**Token hygiene.** Copied file→file and read by a runner script inside the VM, so it never crossed a
shell or a unit property: `systemctl show golden-bake … | grep -c -F "$(cat …)"` → **0**. **The leak
grep on the committed log was itself proven before its zero was believed** — token appended to a
throwaway copy, grepped (**1**), copy shredded, then the committed log's **0** accepted.
**Teardown:** `pct destroy 9100 --purge`; token, runner, build script and in-VM log shredded **after**
the log was copied out; VM powered off; qemu exited; **`qemu-img snapshot -a virgin` restored.**
Evidence: `documentation/tests/golden-0.218.0-2026-08-22/`.
---
## 13. STOP — THE APPROVAL
**Nothing below has been changed. Vouching and the floor are yours.**
### The three Day-0 values, each with BOTH checks
| field | set to | downloadable | selectable |
|---|---|---|---|
| **`golden_version`** | **0.218.0** — sha `8e427869d13eafb71562b77d1535eef6c7f32b4db24f659b988ec6d7db8f478b` | ✔ HTTP 200, 657 026 013 B, and the **downloaded bytes** hash to exactly the bake's reported sha | ✔ offered in the hub dropdown with `data-sha=8e427869d13eafb7…` |
| **`agent_version`** | **0.130.0** — sha `a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3` | ✔ HTTP 200, 14 141 158 B, downloaded sha matches the hub's `data-sha` exactly | ✔ already the SELECTED option — **no change needed** |
| **`min_agent`** | **0.129.0** | — | ✔ already set to 0.129.0 — **no change needed** |
**So the vouch is a ONE-field change this time, and that is the safe direction, not a shortcut.** The
three-field rule exists so a golden is never shipped onto an agent older than it needs. The controller
being baked declares **`MinAgent: 0.129.0` (unchanged)**, the vouched agent is already **0.130.0**,
and `0.130.0 ≥ 0.129.0` — so `agent_version` and `min_agent` are already correct and only
`golden_version` moves. Setting `min_agent` above the vouched agent is the R-216 shape; it is not
happening here.
**If the hub refuses the save, the guard is working.** The R-120 gate on that POST
(`hub/internal/web/configs.go:1165`) refuses a golden older than the newest controller the fleet
reports. 0.218.0 is the newest, so it should pass — and if it does not, fix the cause, never the guard.
**Vouching is reversible:** re-select 0.217.0 and Save. The old package is never deleted by a bake.
### The one line on the floor
**Raising the floor to 0.218.0 is what puts this on both machines — and yes, you will want to.**
`demo-hp` already runs 0.218.0 (deployed directly for the live proof). **`demo-felhom` is still on
0.217.0 and will not move until the floor is raised**, so today it still has both defects. I did not
raise it; that is your call, as is vouching.
---
## 14. WHAT WAS DROPPED, AND OBSERVATIONS
**Dropped: nothing.** Both parts completed in the required order, with the live walk and all seven
red-proofs. The session did not run short.
**Observations, noticed and not acted on:**
- **`demo-felhom` was not touched** and still runs 0.217.0 — so the fleet is deliberately non-uniform
until the floor moves. Its two preserved fixtures were not involved in any step of this session.
- **The `.fab` export has the same R-355 blind spot** and is fixed by the same change — `export.go:600`
filters on the same `db.StackName`. Not separately verified live; the unit tests cover the resolver
that feeds it.
- **`paperless-ngx`'s orphan directory is now a permanent fixture on `demo-hp`** until R-367 is
decided. It is the only reproduction of the pre-fix state left anywhere, which is an argument for
leaving it alone for now.
- **The safety dump still overwrites the unit's own DB dump before renaming it** (R-361, out of scope)
— visible in this session's own evidence, where `romm`'s `db-dumps/` holds both a `pre-restore-*` and
the regular dump.
---
## 15. THE PICTURE — `where-felhom-stands` brought up to date (same day, after the fixes)
Regenerated from `where-felhom-stands.yaml` (the HTML is a build product; the YAML is the source).
**Eight claims re-checked, three moved — all downward.**
| claim | was | now | why |
|---|---|---|---|
| `backup.offsite` | walked | **partial** | "18 snapshots, daily, unbroken" was true on 2026-08-09 and false by 2026-08-21 — the next snapshot after it was put there by hand, twelve days later, and nothing reported the gap |
| `fail.wiped-reinstalled.data` | walked | **partial** | a real reinstall orphans BOTH off-premises tiers: restic silently for 12 days (R-193), and the pre-reinstall PBS archives cannot be opened by the rebuilt box at all (R-366) |
| `backup.fill-warning` | walked | **partial** | the warning fires correctly, but once a day — a filesystem filling at 03:31 is unannounced for ~24 h, and it was watched staying silent while a volume sat at 99% |
**Held, with their notes rewritten:** `backup.tier1` and `recover.byte-identical` now carry the
R-355/R-354 story *and* their fixes; `backup.restore-proof` stays grey for a sharper reason (its
archives are orphaned, not merely untested); `backup.sikeres` gained two fresh instances;
`fail.customer-self-restore` records that R-356 blocks 40 of 53 apps regardless of who is driving.
**`unproven.py --summary` moved, and the checklist asks that it be said:**
**NOT WALKED went from 32 of 55 to 35 of 55.** That is the three downgrades above. Nothing was raised
— the dataset's own rule forbids raising a status here, and nothing needed it.
### A defect found in the page's own toolchain
**The rendered header cited the wrong commits and would have gone on doing so.** `render_stands.py`
parsed `verified_on` from the YAML but had the three commit shas **hardcoded**, so the page printed
the 2026-08-09 commits while the dataset said 2026-08-22. That is precisely the stale-build-product
failure the renderer's own docstring says it exists to prevent. Parsed now, literals gone from both
the script and the output, checked. The count beside it also read *"15 status(es) moved in that pass"*
when 15 was every recorded move ever accumulated; it now separates the two numbers.
**`check_stands.py` was proven able to convict before its OK was accepted:** a claim marked `missing`
flipped to `walked` in a scratch copy fired rule 5 by name (`use.dlna: status 'walked' but NO evidence
document cited`), and the real file still passed. Gates all OK; CI **id=380 / run_number=248**, green.
Confirmed by ID in §11 of the commit trail below.