R-442 CLOSED (controller v0.236.0): "delete my data too" deletes it or refuses; R-465..467 opened

- OPEN-ITEMS: R-442 row removed; R-465 (six remaining Paths.HDDPath readers — audit),
  R-466 (recovery-unit residue after "delete backups"), R-467 (v0.236.0 owes a golden).
- CLOSED-ITEMS: R-442 compressed, reasoning kept, fleet shape now established.
- 00-capability-map: lifecycle row narrowed (Campaign 3 proved remove removes the APP,
  not the data) and re-proven from audits/R442-2026-09-13/.
- STATUS: item 13 in plain language; item 7 closed (ruled 2026-09-02, 09 §3).
- audits/R442-2026-09-13/: live evidence (A, C, D bodies, controls, log window, teardown).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

.gitea/workflows/gates.yml

@@ -92,11 +92,90 @@ jobs:
           git checkout -q FETCH_HEAD
           echo "controller CHANGELOG at $(git rev-parse --short=12 HEAD): $(head -1 CHANGELOG.md)"
 
+      - name: Fetch the app catalog (decoy-coverage reads all four runners)
+        # R-421's meta-gate walks every registered gate across all four repos, so it needs all four
+        # present. Without this it reports the catalog's runner as missing — and a gate that cannot
+        # see part of its subject must not report a pass on it. Same reasoning, and the same fix, as
+        # the two sibling fetches above: give the gate what it needs rather than let it skip.
+        run: |
+          git init -q ../app-catalog-felhom.eu
+          cd ../app-catalog-felhom.eu
+          git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/app-catalog-felhom.eu.git
+          git fetch -q --depth 1 origin main
+          git checkout -q FETCH_HEAD
+          echo "catalog at $(git rev-parse --short=12 HEAD)"
+
+      - name: Classify the push - code or documents (R-404)
+        # ONE RULE, NOT TWO. The pre-push hook exempts a golden-currency CONVICTION on a
+        # documents-only push; if CI did not do the same, a drill night would still produce red CI
+        # runs indistinguishable from real ones, which is R-417 exactly and is half the reason this
+        # change exists.
+        #
+        # CI CANNOT USE A COMMIT RANGE. The checkout above is `--depth 1` of a single SHA, so there
+        # is no history here to diff against — `git diff before..after` would fail, and deepening
+        # the fetch to make it work would slow every run to solve a problem the push event has
+        # already answered. So the file list comes from the push event payload instead, and is fed
+        # to the SAME classifier the hook uses (`--files-from`), so there is one implementation of
+        # "what counts as a document" and not two.
+        #
+        # FAIL CLOSED, EVERY PATH. No payload, no `commits` array, an empty array, unreadable JSON,
+        # a missing classifier — all write `code`, which is exactly today's behaviour. This step can
+        # therefore only ever make CI as strict as it is now, never looser. That is also why it is
+        # safe to ship before it has been observed on a real push: the untested direction is the
+        # safe one.
+        run: |
+          set -u
+          python3 - > /tmp/pushed-files.txt <<'PY' || : > /tmp/pushed-files.txt
+          import json, os, sys
+          path = os.environ.get("GITHUB_EVENT_PATH", "")
+          if not path or not os.path.isfile(path):
+              sys.stderr.write("no GITHUB_EVENT_PATH - the file list is unknown\n")
+              raise SystemExit(0)
+          try:
+              ev = json.load(open(path))
+          except Exception as e:
+              sys.stderr.write("event payload unreadable: %s\n" % e)
+              raise SystemExit(0)
+          commits = ev.get("commits") or []
+          if not commits:
+              sys.stderr.write("the payload carries no commits array - unknown\n")
+              raise SystemExit(0)
+          seen = []
+          for c in commits:
+              for key in ("added", "modified", "removed"):
+                  for f in (c.get(key) or []):
+                      if f not in seen:
+                          seen.append(f)
+          sys.stderr.write("%d commit(s), %d distinct path(s) in the payload\n"
+                           % (len(commits), len(seen)))
+          for f in seen:
+              print(f)
+          PY
+          echo "--- paths the push event reported ---"
+          cat /tmp/pushed-files.txt
+          echo "-------------------------------------"
+          if [ -s /tmp/pushed-files.txt ] && [ -f scripts/push_scope.py ]; then
+            SCOPE=$(python3 scripts/push_scope.py --files-from /tmp/pushed-files.txt) || SCOPE=code
+          else
+            echo "no usable file list - treating this push as CODE (fail-closed)"
+            SCOPE=code
+          fi
+          [ "$SCOPE" = "docs" ] || SCOPE=code
+          echo "PUSH_SCOPE=$SCOPE" >> "$GITHUB_ENV"
+          echo "scope: $SCOPE"
+
       - name: Run the gate entry point
         # The ONLY thing CI runs. No go build, no go test, no linting, no deploy — those are either
         # already reliably run by a person or none of CI's business. The exit code IS the result:
         # no `|| true`, no pipe that could swallow it.
-        run: python3 scripts/repo_gates.py --fast
+        #
+        # A documents-only run that convicts ONLY on golden-currency prints the advisory and stays
+        # green. THE DEBT IS NOT HIDDEN WHEN THAT HAPPENS — three things still carry it: the
+        # advisory block in this run's own log, `STATUS.md`, and the controller repo's golden-notice,
+        # which prints at the moment a release is committed, where someone can actually act on it.
+        # Those are the compensating controls that make this green honest. Every other gate still
+        # fails this job on any push, and golden-currency still fails it on a push touching code.
+        run: python3 scripts/repo_gates.py --fast --scope="${PUSH_SCOPE:-code}"
 
       - name: Alarm on failure
         # THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO

.githooks/pre-push

@@ -16,11 +16,35 @@
 #   * skippable   — `git push --no-verify` bypasses this entirely. That is on purpose: an escape
 #                   hatch that cannot be reached is one that gets removed the first time it is
 #                   inconvenient. USING IT MUST BE STATED IN THE SESSION REPORT.
+#   * ONE GATE IS ADVISORY ON A DOCUMENTS-ONLY PUSH (R-404, 2026-09-01) — golden-currency, and
+#                   only it. On a push whose whole range touches documents, the register, STATUS,
+#                   reports or drill evidence, a golden-currency CONVICTION is printed loudly as
+#                   ADVISORY and does not refuse the push. Every other gate still refuses every
+#                   push, and golden-currency still refuses a push that touches code.
+#                   WHY: the gate never looks at the push — it compares the controller's newest
+#                   CHANGELOG heading against this repo's bake evidence, so it returns the same
+#                   verdict whatever you are pushing. The controller's code is in one repo and its
+#                   register lives here, so EVERY controller change produces a documents-only push
+#                   here; and the push that PAYS the debt (a bake record under documentation/tests/)
+#                   is itself documents-only, so blocking here blocked the cure. `--no-verify` had
+#                   been used thirteen times, each with a recorded reason. This removes the reason,
+#                   not the hatch.
+#                   The scope is decided by scripts/push_scope.py, which fails closed to `code`.
 # The half that is neither per-clone nor skippable is CI — felhom.eu OPEN-ITEMS.md R-168.
 #
 # Measured 2026-08-02 (git 2.47.3): a relative core.hooksPath resolves correctly and the hook's cwd
 # is the repo root whether `git push` is issued from the root or from any subdirectory. The
 # explicit rev-parse below does not depend on that.
+#
+# Measured 2026-09-01 (git 2.47.3, throwaway local remote, probe removed): git hands this hook its
+# ref updates on STDIN as `<local ref> <local sha> <remote ref> <remote sha>`, one line per ref,
+# four whitespace-separated fields. Observed directly:
+#   ordinary push  refs/heads/master <new> refs/heads/master <old>
+#   FIRST push     refs/heads/master <new> refs/heads/master 0000000000000000000000000000000000000000
+#   two refs       two lines, one per ref
+#   deletion       (delete) 0000000000000000000000000000000000000000 refs/heads/side <old>
+# The all-zero cases are exactly why the classifier fails closed: a first push has no range to diff
+# and a deletion has no content, so neither can be exempted.
 set -u
 
 root=$(git rev-parse --show-toplevel 2>/dev/null) || {
@@ -70,8 +94,30 @@ if ! command -v python3 >/dev/null 2>&1; then
     exit 1
 fi
 
-echo "pre-push [felhom.eu]: running scripts/repo_gates.py --fast ..."
-python3 "scripts/repo_gates.py" --fast
+# ── SCOPE (R-404) ────────────────────────────────────────────────────────────────────────────────
+# Read git's ref updates from stdin and ask the classifier what kind of push this is. EVERY failure
+# path here answers `code`, which is today's behaviour — this can make the hook stricter than
+# intended, never looser. The classifier prints its reasoning on stderr, so a surprising verdict is
+# arguable rather than mysterious.
+#
+# STDIN IS CONSUMED EXACTLY ONCE, here, into a variable. A second reader would get nothing and the
+# classifier would answer `code` for a reason that has nothing to do with the push.
+refs=$(cat)
+
+scope=code
+if [ ! -f "scripts/push_scope.py" ]; then
+    echo "pre-push [felhom.eu]: scripts/push_scope.py is ABSENT - treating this push as CODE." >&2
+else
+    # stdout is the verdict word; the classifier's reasoning goes to stderr and is left visible on
+    # purpose, so a surprising verdict can be argued with instead of guessed at.
+    scope=$(printf '%s\n' "$refs" | python3 "scripts/push_scope.py" --prepush-stdin) || scope=code
+fi
+# Anything that is not exactly "docs" takes the strict path. This is the fail-closed hinge: an empty
+# variable, a crashed classifier, a typo and an unexpected word all land on `code`.
+[ "$scope" = "docs" ] || scope=code
+
+echo "pre-push [felhom.eu]: running scripts/repo_gates.py --fast --scope=$scope ..."
+python3 "scripts/repo_gates.py" --fast --scope="$scope"
 rc=$?
 if [ "$rc" -ne 0 ]; then
     echo "pre-push [felhom.eu]: PUSH REFUSED - gates exited $rc. Fix the finding above, or bypass with" >&2

CLAUDE.md

@@ -83,10 +83,22 @@ box**.
 
 **Run `python3 scripts/repo_gates.py` after ANY change in this repo.** It runs every gate —
 `site_gates.py`, `hostinstall_gates.py`, `hub_confirm_gate.py`, `manifest_bearer_gate.py`,
-`reuse_refs_check.py` and `instructions_gate.py` — streaming each gate's own output and exiting
+`reuse_refs_check.py`, `instructions_gate.py`, `golden_currency_gate.py`, `wire_contract_gate.py`,
+`hub_copy_gate.py` and `due_checks_gate.py` — streaming each gate's own output and exiting
 non-zero if any fails. `--fast` selects the gates that touch no network and no container runtime;
 today that is all of them. **A missing gate script is a FAILURE, never a skip.**
 
+`due_checks_gate.py` refuses the push when a dated check in `OPEN-ITEMS.md`'s `DUE-CHECKS` block has
+come due (R-341). **It is not a scheduler** — it fires on the next push, not on the date.
+
+**A gate ships with a decoy test that has been seen to fail (R-421).** A decoy is the LABEL without
+the FACT — a directory with the right name and no bake log, a note whose prose mentions the marker it
+lacks. `scripts/decoy_coverage_gate.py` refuses a new gate that has neither a decoy nor a named
+exemption carrying its row. The four shapes, the 2026-09-01 sweep that fooled 16 of 29 gates, and the
+decoys withdrawn as illegitimate: `documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and
+`felhom-controller/.claude/rules/gates.md`. **Scope is a fact too** — prefer `os.walk` over
+`os.listdir`, and a glob over a hand-maintained list.
+
 `site_gates.py` is a *gate*, not a runner — do not model new work on it;
 `app-catalog-felhom.eu/scripts/catalog_gates.py` is the canonical runner (R-161).
 
@@ -125,8 +137,26 @@ something, not only sessions that touch `documentation/` — which is why it is
   the operator in plain language, and deliberately **not** `CONTEXT.md`.
 - **The capability map** (`documentation/architecture/00-capability-map.md`), if a capability's
   status changed — with its new evidence citation.
+- **`python3 scripts/unproven.py --summary`** — one line per status, and the not-walked total. Run it
+  at the end of any session that shipped, broke or proved something, and **say in the report if a
+  number moved**. It exists because "which claims are unproven?" was answerable only by a person
+  reading a page: a session asked for "the nine grey claims" could not determine which nine and
+  rightly refused to guess (R-326). *Nine was real and answered a different question — it is the
+  count of claims the 2026-08-09 pass DOWNGRADED. Not-walked is 35 of 55 as of 2026-09-01 -- the figure read 32 here for weeks while the tool said 35, so re-read the tool rather than this line.* A status that moves
+  without anyone noticing is how the picture stops being true.
 - **Confirm your own last push's CI run went green, by run ID.** CI emails on failure, which is a
   PUSH signal; this is the PULL check that catches a lost, filtered or unread mail. Quote the run id
-  and its conclusion, e.g.
-  `curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/<repo>/actions/tasks?limit=3"` → match the
-  `head_sha` to your commit. An unchecked green is an assumption, not an observation.
+  and its conclusion. **Use the `jobs` endpoint and match on `head_sha`, never on an id** (R-417,
+  measured 2026-09-01): `actions/tasks` returns `"conclusion": null` for every run, so a session
+  following the old recipe here quotes a conclusion it never read; its `id` is also offset from the
+  `jobs` id for the same run (479 vs 478), and `actions/runs/<n>` takes a JOB id, so `runs/294`
+  cheerfully returns an unrelated job from three weeks earlier. The list is oldest-first — page to
+  the end.
+
+  ```bash
+  T=$(curl -s -u "$U:$P" ".../actions/jobs?limit=1" | python3 -c 'import json,sys;print(json.load(sys.stdin)["total_count"])')
+  curl -s -u "$U:$P" ".../actions/jobs?limit=50&page=$(( T/50 + 1 ))"   # then grep your own head_sha
+  ```
+
+  An unchecked green is an assumption, not an observation — and so is a green read off a field the
+  API never populates.

CONTEXT.md

@@ -14,6 +14,557 @@
 > language, one screen, no identifiers in the prose. Same subjects, different readers; merging them
 > would make one of the two audiences stop reading. `STATUS.md` is also a **view of `OPEN-ITEMS.md`**
 > and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
+## The record told a worse story than the truth for two months, and my own probe is why (2026-09-01, R-429 / R-95 / R-431)
+
+**THREE RULINGS.**
+
+**1. A correct instrument pointed at the wrong subject produces a confident wrong answer, and the
+controls cannot save you.** The 2026-08-31/09-01 spike probed for a directory called `.snapshots` and
+reported *"the safety net cannot be seen from the box"*. Its positive and negative controls were
+sound. **The vendor documents the path as `/.zfs/snapshot`.** So `not found` was true and meant
+nothing, and the register carried "the exposure is unbounded" on the strength of it. Re-probed at the
+documented path: `/.zfs` lists from inside the jail, and **a write into `/.zfs/snapshot` is REFUSED**
+(`dest open …: Failure`) while the identical write to the account home succeeds. **Controls prove the
+instrument works; they say nothing about whether it is aimed at the question.** Cite the vendor before
+asserting a mechanism — the task that set the spike asserted it without a citation, and I did not check.
+
+**2. The real R-95 is much smaller than two months of register text.** The box can delete its own
+**live** repository, but it **cannot write to the daily snapshots of it** — measured, both boxes. So a
+deletion costs at most the data written since the last daily snapshot, and the rest comes back **file
+by file**, one customer at a time, with no effect on anyone else. **Not open-ended loss.** Two limits
+kept honest: a sub-account sees the snapshot directory EMPTY, so per-file recovery is **operator-only
+today** (R-432), and a panel-driven restore rolls back the whole Storage Box. **R-95's ranking is
+Viktor's** — it has been #1 since July on the old story.
+
+**3. Detection belongs on the HUB, not the box, and the threshold is reasoned rather than invented.**
+The event being detected is a box deleting its own backups, so a detector living on that box is one
+the same event can silence. The hub already receives the count and keeps the history.
+**The measurement is the interesting part:** across 12 898 reports (2026-06-05 → 2026-09-01) every one
+of the nine decreases lands exactly on **zero**, and every one predates `stats_known` — they are the
+R-331 shape, a zero meaning *unmeasured*, several carrying a declared `State` that says so outright.
+In the 380-report window where `stats_known` is true there are **zero** decreases. **So observed churn
+gave nothing to calibrate against, and saying so beats inventing a number (R-401).** The threshold
+therefore comes from what retention *can* do: keeping 7 daily + 4 weekly + 6 monthly per group means
+it **cannot halve a total**, while a mass deletion goes to ~0 — hence *more than half, and at least
+five*. Three pre-conditions guard it, each with a scar: `StatsKnown` (R-331), the declared `State`
+(R-204), and run success (R-100, whose lesson lives in that very file). **Acceptance: 9 009 real
+report points replayed → zero alarms.**
+
+## The checks were the one thing nothing checked — 16 of 29 could be fooled by a label (2026-09-01, R-421)
+
+**THE CLASS: an instrument that matches a LABEL rather than the fact it names.** Five instances, and
+every single one was found by accident, by someone looking at something else: R-410 (a `mkdir` turned
+the release gate green), R-400 (seven debug controls answering nothing), R-378 (a status word inside a
+longer sentence), R-419 (a phrase inside prose — including prose stating the marker was absent), R-94
+(a test comparing a constant to itself). **The gates enforce every other rule in this project,
+including the rule that findings must be written down. Nothing had ever checked the gates.**
+
+**The method, and it is the transferable part: write the decoy.** For each gate, construct the label
+without the fact — a directory with the right name and no bake log, a handler case that exists only
+in a comment, a note whose prose mentions the marker it lacks — run the gate, and record what it
+says. **A last column answered from reading is worth nothing**; that is precisely how all five hid.
+
+**Result: 29 scripts, 19 sound, 4 holes left open with rows, 6 that no plausible decoy could be built
+for and are named as UNTESTED rather than called sound.** Sixteen were fooled; ten were fixed the
+same day.
+
+**The largest single cause was mundane and worth remembering: SCOPE IS A FACT TOO.** Eight gates
+decided what to look at with `os.listdir` — one directory level. Every one was green *and correct*,
+because no subdirectory exists today; every one would have gone blind the moment anyone added
+`templates/partials/`, which is an ordinary act. `mojibake` and `docker-v` already used `os.walk`,
+caught the identical planted file, and are the control that proves the cause was the listing and not
+the decoy. **Prefer `os.walk` over `os.listdir`, and a glob over a hand-maintained list of files** —
+three of the four remaining holes (R-423 `PAGES`, R-425 `FILES`, and R-410 itself) are hand-maintained
+lists that silently narrowed as the thing they described grew.
+
+**A decoy nobody would write proves nothing, and saying so is a result.** Five of mine were withdrawn
+as illegitimate rather than counted — an inert Compose `x-` field, a CHANGELOG heading that did not
+actually hide the release, a half-built decoy that left the real call in place, a malformed table row
+that convicted for a structural reason, and several planted files carrying nothing the gate hunts for.
+Manufacturing a finding to fill a row is worse than an honest NO.
+
+**The rule that keeps the answer true:** a gate ships with a decoy test that has been seen to fail.
+`scripts/decoy_coverage_gate.py` refuses a new gate that has neither one nor a named exemption
+carrying its row — and it convicted *itself* the moment it was registered, which is how it got one.
+The 20 gates not yet covered are listed by name (R-426), so the debt is visible and shrinks.
+
+## A guard aimed at the wrong repository trains everyone to bypass it (2026-09-01, R-404 / R-417)
+
+**The ruling was neither option as framed.** The question on the table was whether a documents-only
+push should be subject to the golden-currency gate, and both answers offered — *narrow the gate*, or
+*leave it and write waivers* — argued about the gate. **The gate was never the problem.**
+
+The diagnosis, measured from live source rather than reasoned: `golden_currency_gate.py` **never
+looks at the push**. It compares the controller's newest CHANGELOG heading against this repo's bake
+evidence and returns the same verdict whatever you are pushing — correct for a standing invariant,
+wrong as a push gate. And `controller_gates.py` had **no golden-currency entry at all**. So the
+repository where a release actually happens never checked, while the repository that cannot create
+the debt enforced it on every push. **18 of the last 24 pushes here touched no code** — measured, and
+the classifier built for this agrees exactly — most of them by construction, because the controller's
+code lives in one repo and its register, architecture and status live in this one. Worse: **six of
+those eighteen were bake records**, so the push that pays the debt is itself documents-only and the
+gate was blocking its own cure. `--no-verify` had been reached for thirteen times, each with a
+recorded reason, which is what a correctly-bypassed guard produces.
+
+**The ruling: block the push that can create the debt, notify the push that cannot.** The gate's
+logic, exit codes and wording are byte-identical; only the consequence changed, for one gate, on one
+kind of push, with a loud ADVISORY block so nothing goes quiet. A notice now fires in the controller
+repo at the moment a release is committed — advisory in every case, because at that moment the golden
+legitimately cannot exist yet.
+
+**Why not the waiver the gate's own docstring prescribes.** That clause was written for *a release
+nobody wants a golden for*. The case that actually occurred was *a release we did want a golden for,
+on a night the drill runbook forbade baking*. A waiver would have recorded a lie.
+
+**What did NOT change, and must not be presumed:** nothing gates the **vouch**. A baked-but-unvouched
+golden still passes both the gate and the new notice — R-242's remaining half, still open, for the
+unchanged and forced reason that the vouched version exists only in the hub's database.
+
+## Skills cover the PROCESS domain too, from named MIT sources with named exclusions (2026-08-25)
+
+**[RULING] `felhom.eu/skills/` now holds two kinds of skill and the distinction is deliberate.** The
+four originals — `felhom-app-catalog`, `felhom-build-deploy`, `felhom-testing`, `felhom-ui-design` —
+are **product-domain**: what the system is and how to change it. The five added 2026-08-25 —
+`felhom-evidence`, `felhom-diagnosis`, `felhom-plain-language`, `felhom-handoff`,
+`felhom-doc-authoring` — are **process-domain**: how work is judged, diagnosed, written up and handed
+over. The gap they close is that two rules this project had already paid for lived only in
+conversation, so Claude Code never read them.
+
+**The process skills are ADAPTATIONS of two MIT collections (`mattpocock/skills`,
+`backnotprop/pstack`), not copies, and neither repo is installed, vendored or depended on.** Six
+pieces of upstream material were excluded on purpose — most importantly *"stop asking and proceed on
+reversible work"*, which reasons from *code is cheap and revertible*; Felhom's work reaches live hosts
+and one real customer's data, where that premise is false. **The exclusions and their reasons live in
+`skills/SOURCES.md`**, so a session that finds the upstreams sees a decision and not an oversight.
+
+**One rule, one home — this is load-bearing, not tidiness.** A rule that already lives in another
+skill or in `documentation/runbooks/workspace-CLAUDE.md` is POINTED AT by name, never restated: two
+copies drift and the reader cannot tell which is current. So `felhom-diagnosis` sends you to
+`felhom-testing` for the red-proof, and `felhom-evidence` cites standing rules 2 and 3 in one line
+each. **`skills/felhom-doc-authoring/SKILL.md` is where that rule and the 150-line limit are written
+down** — read it before adding or editing any instruction file.
+
+**Validation: `python3 scripts/check_skills.py`.** `install_skills.py` globs `skills/*/SKILL.md` and
+never reads the file, so a skill missing its `description` installs perfectly and then silently never
+loads. The glob is the registry — **a new skill directory needs no registration, and no manifest or
+index may be added.** What the checker CANNOT see is whether the model actually reaches a skill when
+it should; that is behavioural, decided by the `description` wording, and it is checked by the
+operator typing a probe phrase, never claimed from a green run.
+
+## Cooldown GRAIN is allow-listed, never inferred from the payload (2026-08-23, R-389)
+
+**[RULING] Per-app cooldown is a NAMED REGISTER (`perAppCooldownEvents`), not a rule of the form "if
+the details carry a `stack_name`, split per app".** The backup family's cooldown is coarse **on
+purpose** — R-97a and R-182 exist so one full disk sends one digest rather than one mail per app.
+
+**[FACT] The payload-shape rule would have been wrong, and provably so.** `crossdrive_failed` is
+severity `error`, reaches the operator leg, and carries `stack_name` through **`CrossDriveDetails`**,
+a different struct from `AppDetails`. Anything keying off the field would have split it silently.
+**That is why `cooldownStackSuffix` takes the EVENT TYPE as well as the details** — an asymmetry with
+`cooldownTierSuffix`/`cooldownRunSuffix`, and the asymmetry IS the safety property: `tier` and
+`run_id` appear only on types that want that grain; `stack_name` does not have that property.
+
+**[RULING] `app_start_failed` gets per-app because it has NO DIGEST.** There is no `apps_down_run`
+summarising a scan the way `backup_run_failures` summarises a run, so per-app is the only grain that
+does not lose alarms. **The hour did not change** — the grain was the complaint, not the length.
+
+**[FENCE] The fenced act is adding an entry to `perAppCooldownEvents` for a type whose family has a
+digest or a deliberately coarse cooldown.** Reading it is fine.
+
+**[MEASURED] Burst volume, so it is a number and not an impression:** three apps in one scan →
+**3 attempted, 3 sent, 0 suppressed**. Reference box has 8 deployed apps, so a total outage is 8
+mails. It scales linearly and has no ceiling; the reopening condition is a box big enough that a
+total outage is unreadable, and the answer there is a digest, not a wider cooldown.
+
+## A finding recorded only in REPORT.md has a lifetime of ONE SESSION (2026-08-23, R-389 / gate 11)
+
+**[FACT] This cost a day.** The cooldown-grain defect was measured live on 2026-08-23, written under
+`## Observations` in `REPORT.md`, and written **nowhere else**. `REPORT.md` is overwritten every
+session by this project's own convention. It had no register row and had to be re-derived the next
+day. Same shape as R-341.
+
+**[FACT] The instruction invited it.** `PROMPT-TEMPLATE.md` §15 asked for observations *"documented,
+NOT acted on"*, and "documented" was satisfied by the paragraph. **The template was the defect**, and
+it was corrected in the same session that built the mechanism.
+
+**[MECHANISM] Gate 11 (`scripts/observations_gate.py`).** Every numbered item in a `REPORT.md`
+observations section must carry `FILED: R-NNN` (which must resolve) or `NOT-A-FINDING: <reason>`. It
+REFUSES; it does not warn. It PASSES quietly when there is no observations section, deliberately — a
+gate that taxes every push gets disabled within a week.
+
+**[RULING, and it contradicts the task that commissioned it] A bare mention of an R-number does NOT
+satisfy the gate.** The specification said an item may "cite an R-NNN that resolves" — **that rule
+would have passed the very item the gate was built to catch**, because the lost item cited `R-182` as
+an *analogy*. No parser can tell citation-as-precedent from citation-as-filing by reading prose. The
+marker is explicit for that reason, and the discrepancy is recorded in the gate's docstring rather
+than quietly resolved.
+
+**[GOTCHA] Registered in three runners, not four.** `catalog_gates.py` has no shared-gate mechanism —
+`run_gate` joins against its own `scripts/` and the loop appends `--all` to every gate — so
+registering there needs its contract widened. Filed as **R-391** rather than left as a sentence,
+which is this session's whole lesson.
+
+## The severity a controller sends is the HUB's vocabulary, and getting it wrong deletes the alert (2026-08-23, R-329 / R-387)
+
+**[RULING] The set is exactly `{info, warning, error, critical}`.** The hub **coerces anything else to
+`info` at ingest**, and `severityNotifies` drops `info` **before both** delivery legs. So a severity
+outside the set means: stored, `200` returned, dashboard shows it, **e-mailed to nobody**.
+
+**It shipped twice** — `DiskAlertKind.Severity` until controller v0.215.0, `app_start_failed` until
+v0.223.0. Measured on the live hub DB: **91 `app_start_failed` events stored all-time, ZERO
+`notification_log` rows** before 2026-08-23.
+
+**[MECHANISM, because a comment already recorded this lesson and it happened again]** An AST walk over
+the whole controller pins every emitted severity. **grep cannot do this job** — `"warn"` is a
+legitimate *healthcheck status* in `internal/monitor` and `internal/selftest`; the 2026-08-23 sweep hit
+nine such strings and exactly one defect. The walk cannot follow a variable, so the six call sites that
+pass one are **registered by name with the values each can take**; a new dynamic site fails the test.
+**An unlisted limit is not a limit, it is a hole.**
+
+**[RULING] The hub's coercion STAYS and now SPEAKS (hub v0.107.0).** A rejected event is a *lost*
+event, and losing an alarm is worse than mis-routing one — that is why it was a coercion. The fix is
+that it logs `WARN` naming customer, type and value. **The guard that should have caught this class sat
+downstream of the rewrite and was structurally blind to it** — note the asymmetry it lived beside: an
+unknown event *type* 400s loudly, an unknown *severity* was silent.
+
+**[RULING] `dispatcher.go`'s `unrecognized severity` branch is KEPT, on evidence.** `cmd/hub/main.go`
+wires `ProcessEvent` **directly** as the `monitor.EventNotifyFunc` for three checkers that never pass
+the ingest handler. For those it is the only severity guard there is.
+
+**[RULING] `app_start_failed`: operator always, customer OFF by default.** `processOperator` consults
+only `operatorOn`, the address and a 1-hour cooldown — **never** customer preferences — so one word
+fixed the operator leg and left the customer leg where the ruling wanted it. It is deliberately **not**
+in `operatorOnlyEvents`: that would make the new toggle visible, flickable and structurally incapable
+of delivering.
+
+**[GOTCHA, now load-bearing] The operator cooldown key carries no app identifier**
+(`customerID:eventType`), so **only the first app-down per hour e-mails the operator**; the rest are
+logged `suppressed`. Harmless while the event was undeliverable. Not any more. R-182's shape, filed.
+
+## "The customer stopped this" is a RECORD, never an inference (2026-08-23, R-386)
+
+**[RULING] Ask `DesiredState`, do not read the container state.** `aggregateState` folds `StateExited`
+into the stopped counter, so an all-down stack returns `StateStopped` whatever killed it — an
+out-of-band stop and a customer's Stop are byte-identical on the Docker side. Measured: `privatebin`
+stopped out of band, nine scans, **zero events, zero banner**, while a comment claimed it "still
+alerts".
+
+**[RULING] The tri-state, and the third value is the whole safety property.** `Stopped` → no alarm.
+`Running` → **alarm**. **Absent → UNKNOWN, keep the old behaviour, AND announce it.** Reading absent as
+"nobody asked" would, on the first cycle after upgrade, e-mail about every app any owner ever
+deliberately stopped — fleet-wide, from a field that predates the intent being asked of it. **The
+backfill cannot help: it seeds `Running` only from an observed-UP reading**, so anything stopped at
+upgrade time stays unknown — exactly the ambiguous population.
+
+**[MECHANISM] `AppRunState.IntentUnknown` + an INFO line naming the apps**, on the heartbeat cadence.
+An operator must be able to ask *"how many apps am I blind to?"* and get a number. **A rule without a
+mechanism is a wish.** Measured on `demo-hp`: **0 of 8** deployed apps had an absent intent.
+
+**[FENCE] Adding a `DesiredState` WRITER is the fenced act; reading it anywhere is fine.** Twelve of
+`StopStack`'s fourteen callers are machines, so recording intent in the primitive would make a nightly
+backup indistinguishable from the customer pressing Stop.
+
+**[FENCE] `failedRestart` must still lift a `Stopped` intent** — the quiesce loop stops stacks by the
+same path a customer does, and one it stopped and could not restart must alarm whatever the intent
+says. Removing that term re-opens F-CRIT-1.
+
+## Two settings toggles each governed two alarms, and the labels named one (2026-08-23)
+
+„Lemez figyelmeztetés (90%+)" also wrote `disk_critical` — the drive-is-**failing** alarm. Split into
+four honest toggles; **12 → 15**. **[RULING] The migration, not the split, was the risk:** a save whose
+event **set** is unchanged now stores the **existing slice verbatim**, so byte-identity is by
+construction. Without that guard the defaults case reorders — the red-proof caught it. Legacy compound
+form names are still read, so a stale browser tab cannot drop a key.
+
+## Customer notification model — DIRECTION, recorded and deliberately NOT implemented (2026-08-23, R-388)
+
+The operator's framing, verbatim: *"A customer should be notified only about things they can act on or
+are responsible for… **A failed backup is our incident, not theirs.**… The subscription should feel
+like being looked after, not like being on call."* Full entry at
+`documentation/architecture/08-alarm-ladder.md` §8, marked **not current behaviour**. The settings page
+grew 12 → 15 toggles in one session and grows by one per detector — that growth is the argument.
+**Nothing in this session implements it.**
+
+## App data placement is a DECISION, and it had never been written down as one (2026-08-22, R-376)
+
+**Recorded here on 2026-08-22 to give an existing decision a home. It is NOT new, and this entry is
+not its date** — the choice is older and its original date is not on record. That is itself the
+finding (R-376).
+
+**The decision.** App data is placed **per volume, not per app**. `.felhom.yml` classes each volume
+**hot** — database, config, cache — which goes to fast storage inside the guest and is **enforced**;
+or **bulk** — media and files — which may live on slow attached storage. A photo app's database stays
+on the SSD while its blobs go to the USB drive. Recorded shape:
+`architecture/01-topology-and-trust.md:150-152`, now marked **[DESIGN]**.
+
+**What follows from it, and is a consequence rather than a separate choice:** 40 of the 53 catalogue
+templates declare no configurable data path, because they are all-hot apps. The 13 that do are the
+media and library apps. Stated as **[FACT]** at `07-backup-architecture.md:296-299`.
+
+**What was rejected, so the same proposal does not return.** Moving named-volume (all-hot) app data
+onto the default data drive was proposed in `SPEC-app-data-placement-2026-08-21.md` §5 and is
+**rejected**: it would move hot data onto storage this design classes as possibly-slow, and would put
+it outside the guest vzdump that currently protects it (`01-topology-and-trust.md:154-156`). The spec
+carries the correction inline.
+
+**Why this entry exists at all.** The decision was real, implemented and load-bearing, and lived in a
+single unmarked bullet. Between 19 and 22 August it was called a defect in four places (R-370),
+because a reader met a marked `[FACT]` beside an unmarked choice. **A decision nobody wrote down as a
+decision is one somebody will eventually report as a bug.**
+
+## Read the architecture folder BEFORE calling something a defect (2026-08-22, R-370 / R-369)
+
+**Between 19 and 22 August the reviewing side called a documented architectural decision a defect, in
+four places.** The decision is in `architecture/01-topology-and-trust.md:150-152`: `.felhom.yml`
+classes each volume **hot** (DB/config/cache → fast storage, **enforced**) or **bulk** (media/files →
+may be slow). The 40 catalogue templates with no `HDD_PATH` are all-hot apps — their data is app
+internals and belongs inside the guest, on its own volume so it cannot fill the OS
+(`felhom-agent/configs/build-golden.sh:29-40`). The 13 with a path are the bulk apps. **There is no
+choice being denied**, and the deploy page has been telling the customer exactly this all along:
+*„A kiválasztott meghajtón az alkalmazás fájljai (média, dokumentumok) tárolódnak. Az adatbázis a
+gyors belső SSD-n fut"* (`deploy.html:624-625`).
+
+**Why it happened, mechanically: the register and live source were read; `documentation/architecture/`
+was not.** Nothing in `PROMPT-TEMPLATE.md` §4 required naming the architecture document for the area
+being touched — item 4 named only the controller module map, and the S-1 rule governs *updating* a
+design doc at the end, not *reading* one at the start. **Fixed in the same session:** §4 now carries
+the file→area map, the ordering *architecture = reasoning, register = work, source = truth*, and the
+test **"is what I am about to call a defect something we chose?"**
+
+**Corrected in place, not rewritten** — `SPEC-app-data-placement-2026-08-21.md` and R-352 carry
+`[CORRECTED 2026-08-22]` blocks that say what they got wrong; a document that quietly changes its mind
+teaches nobody.
+
+**And the sibling failure, R-369: there are two registers.** `OPEN-ITEMS.md` calls itself the single
+source of truth; `ROADMAP.md` also holds open work. 72 `R-` ids live only in ROADMAP, 29 of them not
+marked done. **R-107 — the off-site restore never unpacking its own volume tars — was filed there on
+2026-07-28, one day after OPEN-ITEMS was created, and never entered the register. It was rediscovered
+from scratch 25 days later by the 2026-08-21 drill and shipped as R-354.** The template now says
+plainly that a ROADMAP row alone does not satisfy the filing rule.
+
+## Box REACHABILITY is a separate signal from box FILL — and the cadence difference is deliberate (2026-08-18, R-339, hub v0.106.0)
+
+Both off-site checkers now carry two independent signals, and conflating them is the mistake to avoid:
+
+- **FILL** — how full is the store. Escalation-only (one emit on a band rise), and **a degraded or
+  missing reading drives NO band transition**, because missing ≠ 0%. That rule is unchanged by
+  v0.106.0 and must stay unchanged: it is what stops a dead endpoint from reading as an empty one.
+- **REACHABILITY** — can the hub see the store at all. Counted in consecutive failed fetch windows,
+  emitted past a threshold (default 3 ≈ 30–45 min) with a paired all-clear.
+
+**Why reachability REPEATS rather than escalating once.** `emitFill`/`emitOversub` fire once on a band
+rise and stay silent while the condition persists, which is right for a capacity trend. Applied to
+reachability it would produce exactly ONE mail at roughly minute 30 of a nine-hour outage — and one
+mail is missable, which is the whole failure being fixed. So the unreachable event fires on every
+failed window past the threshold and relies on the dispatcher's 1-hour per-type operator cooldown to
+become an hourly "still blind" heartbeat. **If you find yourself "fixing" this back to the band shape,
+this paragraph is why not.** The cost is a `suppressed` notification row every ~15 min during an
+outage — honest bookkeeping.
+
+**Two boundaries worth holding in mind.** `ErrUsageUnsupported` is *not* blindness — an ep0 on an old
+tenantsync answers "no such op", which means we reached it; the counter is not advanced, or the alert
+would fire for days on a healthy pre-update box. And the reachability read rides ep0's **local API
+daemon**, not the HTTPS proxy on 8007 — so it would have shown green throughout the 2026-08-18 outage,
+which is R-340 and is the honest limit of what this watches.
+
+## The managed floor now tracks the vouched golden — and that is rule 2 ARRIVING, not an exception (2026-08-18, R-343)
+
+Since 2026-08-18 12:36:58Z the global managed controller floor
+(`hub_settings.min_controller_version`) reads **0.216.0**, equal to the vouched golden. Read it from
+the store (`GetGlobalMinControllerVersion`, `hub/internal/store/store.go:1751`), never from the form.
+
+**Do not read the preceding gap as drift.** `publish-train-rules.md` rule 1 is *manifest before
+floor* and rule 2 requires the floor field saved **LAST, in a separate save**, because the DB row
+overrides the env floor and acts on the next report cycle. The floor therefore sits behind the newest
+controller **by design** for as long as it takes to bake and vouch a golden — and it is raised
+afterwards. Floor-equals-golden is the state that ordering is meant to arrive at; a session finding
+the floor behind should check whether a vouch is outstanding before calling it a defect.
+
+**The raise is not free, and this is the part worth remembering.** Rule 2's "acts immediately" is
+literal: nine seconds after the save, `demo-felhom` auto-updated 0.214.0 → 0.216.0
+(`controller_updated` 12:37:07Z, `controller_started` 12:37:12Z, no error events). So a floor raise is
+a fleet action, not bookkeeping — plan it as one. `ResolveManagedFloor`
+(`hub/internal/store/store.go:2068`) is what makes it safe: it holds the floor entirely when it
+exceeds the golden (R-216), and per-box when that box's agent is below the manifest's `MinAgent`.
+
+## A name must differ from its neighbours on a stem, not on a noun (2026-08-13, R-323)
+
+The third near-homograph is renamed: the five-word phrase that proves an account owns the box being
+bound is „**Tulajdonosi jelmondat**". It was „Visszaállító jelszó" — one word from the „Visszaállító
+kód" retired days earlier, and false besides: **the phrase restores nothing.**
+
+**The rule this instance yields, and it generalises past this secret.** Both obvious replacements were
+rejected, and the reasons are the transferable part:
+
+- **Do not name a secret after the ACT it is used in, when another secret on the same screen is named
+  after the same act.** The other factor on that page is the „**Párosító kód**". „Összekötési jelszó"
+  would have left the two things a customer types in one sitting separated only by *kód*-versus-
+  *jelszó* — **structurally the very trap being removed**. A name that matches the page's verb feels
+  right and is the wrong axis.
+- **Do not name a secret after a THING that already has a password.** „Fiókjelszó" collides with the
+  dashboard login, i.e. trades one homograph for a worse one.
+- **Separate on the STEM first, the noun second.** Beállító / Helyreállítási / Párosító / **Tulajdonosi**
+  — four distinct stems; *kód* / *jelmondat* is the second axis, not the first. A pair distinguished
+  only by its noun is what failed twice.
+
+**Naming is not function, and the pin says so** (`TestSelfBindPassphrase_StillAcceptedAfterTheRename`
+drives the real handler with human-messy spacing). **Nothing printed was stranded** — no customer
+document names the phrase; it is delivered out-of-band, so the description in the hint sentence is
+what identifies it, which is why that sentence was kept and extended rather than trimmed.
+
+## A guard is worth what its positive control is worth (2026-08-13, R-324)
+
+The hub's customer copy came under a guard for the first time — every customer e-mail and both binding
+pages, the sentences a customer reads *before* they have seen any box screen. Nothing had ever looked
+at them, in either repo.
+
+**The part worth carrying: the selftest found a defect in the guard itself on its first run.** The
+plant→convict→remove→pass control convicted its own step-3 case (a phrase inside a comment) because
+the synthetic source was named `<selftest>` and comment-stripping keys off the `.go` extension. **A
+guard that had shipped without a control would have been quietly over-broad**, and the first person to
+write an honest explanatory comment would have been convicted by it.
+
+**And on sharing a word list across two repos:** the vocabulary lives once, in `scripts/`, the same
+home both repos already consume gates from. The consumer that could not be edited that evening is
+**drift-checked rather than duplicated** — the gate reads the other gate's literal and fails on
+divergence. That is a scaffold with a register row (R-325) to remove it, not a design; **two copies of
+a word list is the R-299 defect waiting to happen**, and a drift check is the cheapest way to refuse
+the copy without making a cross-repo edit at the wrong moment.
+
+## Suppress an alarm on the machine's own declaration, never on an inference — and check for the second door (2026-08-13, R-321)
+
+A box whose reporting is deliberately switched off announces it (`health.status = "disabled"`) and
+then goes quiet by design. The hub stored that, rendered it, and **alarmed on it anyway**, because the
+staleness verdict was computed from report AGE alone.
+
+**Three things worth carrying:**
+
+1. **The discriminator must be the box's own last word.** That is what made this a suppression rather
+   than a guess — and the standing rule is right that suppressing on a guess is worse than the false
+   alarm it removes.
+2. **RECORD the state; do not delete it.** The `blocked` precedent deletes, and `GetState` then
+   returns `""` — which is not `"down"`, so **`CheckBackupDeadlines` would have gone on alarming from
+   a second function.** One fix, two doors; R-195's shape returning through the one nobody looked at.
+   *Ask of any suppression: which OTHER checker reads this state, and what does it see now?*
+3. **A clock must restart at the event that ends the suppression**, not at the last observation before
+   it began. Timing from the pre-disable report would fire an instant stale/down for a quiet period we
+   requested. Re-entering the new-customer branch gives a first observation instead of a recovery
+   event.
+
+**The limit is stated in the code, not hidden:** a box re-enabled that then fails to report keeps
+being suppressed, because the hub's view changes only when a report arrives. **That is why the state
+is made visible** — an operator who re-enabled a box and still sees `disabled` is being told it has
+not come back.
+
+**And a test lesson:** red-proof 2 revealed that scenario A passed on its state assertion alone even
+with the suppression deleted, because it seeded the customer *already* disabled and the new-customer
+branch swallows the first observation. **A test that cannot see the alarm it exists to prevent is not
+a test** — it now observes the machine healthy first.
+
+## A number everyone repeats is a claim, and claims get checked (2026-08-13, R-326)
+
+"The nine grey claims" was repeated across sessions and **exists nowhere**. A session asked to report
+on them could not determine which nine and declined to guess; that refusal was correct and is the
+finding.
+
+**Nine is real and answers a different question: it is the count of claims carrying
+`verdict: downgraded`** — the ones the 2026-08-09 pass *lowered*. The count of unproven claims is
+**32 of 55** (`walked` 23, `partial` 14, `built` 14, `missing` 4), and of those 32 only **6** cite an
+evidence document.
+
+`scripts/unproven.py` makes it a command, wired into the end-of-session checklist. **The generalisable
+part: a picture built for a person to read is not a picture anyone can check.** The dataset carried the
+statuses all along; what was missing was one command that answers the question the operator actually
+asked. Its first run found a stale claim (R-327) — which is the argument for having built it.
+
+## A fact the boxes send and the hub cannot read is a future false green — and one now has a reader (2026-08-13, R-319)
+
+**The first of R-264's twenty-one unconsumed facts is read.** The agent emitted `guest_net` on every
+heartbeat for **twenty-three days** (agent v0.92.0, R-54, 2026-07-21) while the string occurred
+**nowhere** in `felhom.eu/hub/` — stored as raw text in `report_json`, reaching no check, no alarm and
+no screen. Hub v0.104.0 models it: `parseGuestNet`/`guestNet` in `web/hosts.go`, rendered as the
+host-detail **Guest network** card.
+
+**The rule this instance teaches, and it generalises past guest networking:**
+
+- **Read the fact that carries the RISK, not the fact that is easiest to render.** Here that is
+  `heals_last_hour`, not `state`. A guest the watchdog keeps repairing is healthy at every instant
+  anyone looks and is nevertheless failing — rendering the state alone gives it a green tick, which is
+  the failed-disk-drawn-as-a-healthy-empty-disk shape wearing different clothes.
+- **Decode the field that decides the question, not five of the eight.** `heal_succeeded` was added
+  mid-build for exactly this: six FAILED repairs is a guest that is down, six successful ones is a
+  nuisance, and the count alone cannot tell them apart. **R-260 is the standing warning** — a hub
+  decoder three fields short of the agent dropped the one its checker existed to answer.
+- **Three absences, three sentences.** Agent too old / capable agent silent / guest state not asserted.
+  All render unknown; conflating them is how an operator starts ignoring a card. The state switch is an
+  **allow-list**, so adding a state to the agent can never silently paint it green here.
+- **REMOVING an allowlist entry is what proves a reader is wired.** An allowlisted tag is *skipped* by
+  `wire_contract_gate.py`. Re-labelling the eight `guest_net` entries would have left the new reader's
+  own fields unchecked while the gate reported coverage. Deleting them moved checked tags **182 → 190**
+  and skipped **88 → 80** — the positive control that the wiring is real, independent of any test.
+- **A card is not an alarm, and building the card first is deliberate.** No email was added: the
+  incident behind this (a killed `dhclient`, tunnel down 1 h 15 m, nobody told) was a **visibility**
+  failure, and with zero repairs ever recorded on either demo box any alarm threshold would be invented.
+  **The visible count is what will supply the threshold.**
+
+**Where the twenty stand** (R-264, counts measured from the allowlist rather than estimated):
+**8 read · 5 deliberately not consumed · 1 redundant · 6 still owed a reader.** The gate grew a **third**
+entry kind — `not consumed, deliberately` — because an undecided fact and a decided one must not read
+alike, and *"arguably owed"* had been carried for five days as though it were a decision.
+
+**Two findings came out of deciding the last one rather than from building anything:**
+
+- **R-321** — `reporting_disabled` is *redundant* (`health.status = "disabled"` travels beside it, is
+  decoded, and IS rendered), **but `StalenessChecker.Check` is age-only** (`monitor/staleness.go:88+`,
+  sole skip `IsCustomerBlocked`), so a deliberately-silent box still alarms `node_stale` then
+  `node_down`. **Decoding the flag would have felt like progress and left the alarm firing** — the fix
+  belongs in the checker, which already holds the status it needs.
+- **R-322** — `retrieval_promise_gate.py` has **never scanned the hub**, which composes every customer
+  e-mail, i.e. the copy read *before* any box screen. A hand scan with its own four stems returns zero,
+  so it is a scope gap rather than a live defect — recorded before someone writes the first hub-side
+  retrieval promise assuming the guard has them covered.
+
+## One secret in two situations keeps its NAME and changes its SENTENCE — and the mail must name the page the machine shows (2026-08-13, R-295 hub half)
+
+The box side shipped 2026-08-10; the hub half was dropped twice and was the last place „Visszaállító
+kód" survived — in the **mails**, the one surface a customer reads *before* they see any screen. It is
+now „**Beállító kód**" everywhere; the ten-word escrow code stays „**Helyreállítási kód**"; no
+dashboard-code mail names both.
+
+**The mechanism worth carrying: the mail names a PAGE, and which page depends on state only the hub
+knows.** `ReissueForReenroll` sent the *reset* mail, directing the customer to „Elfelejtett jelszó" —
+but a rebuilt box has no password, so the controller computes `reset := s.authEnabled()` → false
+(`felhom-controller/controller/internal/web/claim.go:279`), renders „**A szerver beállítása**", and
+serves **no login page at all**. The named route was not on their screen. **The two situations are
+distinguishable only at the hub, because the hub chose which call site fired** — so the fix is a new
+`EmailKind` (`reenroll`), not a reworded shared template. A `default` branch falls back to the
+first-setup mail, which is the safe direction and is pinned so a future reader knows it was chosen.
+
+**The re-enrol mail deliberately says nothing about apps or backups.** A clean-slate reinstall is
+precisely where that reassurance could be false; `TestFormatClaimEmail_ReenrollPromisesNothingAboutTheData`
+guards the CLAIM rather than one phrasing of it.
+
+**Naming is not function, and the pin says so:** `TestReenrollSplit_ChangesTheMailNotTheSecret` — the
+generation still advances, the stored bcrypt hash still verifies the emailed code, no plaintext is
+persisted. Box-side companion: the controller's `TestResetCode_StillAcceptedOnTheSetupPage`.
+
+## Evidence dies in the INTERMEDIATE revert, not the final one (2026-08-13, R-320)
+
+Twice in three days, on the same box at the same point, a phase's logs were destroyed by a revert to
+`virgin` *between* phases (drill 2026-08-12 Phase A→B; R-316 2026-08-13 Part 1→2). Both times the
+golden-bake runbook's existing *"scp the log OUT first"* was applied to the **final** teardown and not
+the middle one. Both times the conclusions survived on luck.
+
+**Standing rule 5** now: evidence comes off at the end of the phase that produced it, before any revert
+— **the pull is the LAST ACT OF THE PHASE**, not a step to remember later. Four homes:
+`workspace-CLAUDE.md`, `runbooks/target-selection.md`, `RUNBOOK-rehearsal-v3.md`, `PROMPT-TEMPLATE.md`
+report item 8. **And the already-gone case is documented rather than improvised:** say so plainly and
+reproduce the finding independently.
+
+**A corollary this session hit immediately:** the second incident was recorded in `REPORT.md`, which
+every session overwrites — writing tonight's report would have **destroyed the record of a destroyed
+record**. It was preserved to `audits/REPORT-r316-installer-v1.28.0-2026-08-13.md` first. *A durable
+citation may never point at a file whose contract is to be overwritten.*
 
 ## Material retained is not history recoverable — ask the three questions separately (2026-08-12)
 

REPORT-agent-transport-leak.md

@@ -0,0 +1,246 @@
+# REPORT — R-344: the agent's leaked PBS connections, fixed and proven on both boxes (2026-08-20)
+
+**Outcome: the fix works, measured three independent ways; ep0 is back to its baseline of 17 file
+descriptors from 415; and 0.130.0 is now published and vouched.** Both demo boxes run the **byte-exact
+published artifact**. R-347 is CLOSED. Two new findings came out of the release itself — **R-349**
+(the fleet was briefly running a different binary under the same version name) and **R-350** (I printed
+the hub password into the transcript; rotation is your call).
+
+## 1. Confirmed baselines
+
+| repo | in | out |
+|---|---|---|
+| `felhom-agent` | `f17ed11` — v0.129.0 | **`ede49b6`** — 0.130.0, UNRELEASED |
+| `felhom.eu` | `9299f85` | docs + register only |
+
+Both clean and equal to `origin/main` before each build. Agent version in: 0.129.0 on both boxes.
+Out: **0.130.0 on both**, confirmed from the hub, not from the boxes' own `--version`.
+
+## 2. The diff
+
+| file | symbol | change |
+|---|---|---|
+| `internal/httpx/transport.go` | **new package** | `DefaultIdleConnTimeout = 90s`; `NewTransport(tlsCfg, idle)` — fresh transport per call, `<= 0` means **use the default, never "no timeout"** |
+| `internal/httpx/transport_test.go` | new | zero/negative → default; the constant is read off `http.DefaultTransport`; freshness; TLS config preserved |
+| `internal/pbs/client.go` | `Config`, `NewClient` | `IdleConnTimeout` field (tests only); transport via `httpx` |
+| `internal/pbs/client_leak_test.go` | new | Scenarios A/C + the production-default pin |
+| `internal/hub/client.go` | `NewClient` | via `httpx` — **consistency only, did not contribute to the leak** |
+| `internal/proxmox/client.go` | `NewClient` | same |
+| `cmd/felhom-agent/main.go` | `version` | 0.92.1 → 0.130.0 (ldflags default) |
+| `REUSE.md`, `CHANGELOG.md` | — | `httpx.NewTransport` entry; the release note |
+
+Commit `ede49b6` on `main`. `grep '&http.Transport{'` now matches only `httpx` itself.
+
+**A check made before trusting the fix:** `doBody` already reads the body to completion and closes it,
+so the connection genuinely reaches the idle pool. Had it not, the idle timeout would have been the
+wrong fix entirely.
+
+## 3. Tests and the red-proofs
+
+`go build ./... && go vet ./... && go test ./...` → **30 packages, 0 failures.**
+`python3 scripts/agent_gates.py` → **all 4 gates OK.**
+
+The leak test counts connections **server-side** and models what `pbsTargetsFromPVE` does — build a
+client, use it once, drop it. It deliberately does **not** assert `err == nil` or that a field holds a
+value; both were true of the leaking code (the R-224 lesson).
+
+| red-proof | mutation | seen failing with |
+|---|---|---|
+| 1 — the fix | remove `IdleConnTimeout` | *"abandoned pbs.Clients: after 5s the server still holds 5 open connection(s), want 0 (5 dialled in total)"* — the count is in the message, so it cannot be a timeout with another cause |
+| 2 — the worse fix | `DisableKeepAlives: true` | **the leak test PASSES.** Caught only by `TestPBSClient_KeepAliveStillReuses`: *"3 sequential requests over 3 connection(s), want 1"* |
+
+**Red-proof 2 is the load-bearing one: Scenario A alone would have accepted a fix that made the problem
+worse** — no leak, at the price of a fresh dial for every one of ~40,000 daily requests. Both mutations
+reverted, tree re-verified clean.
+
+## 4. §4 quoted, beside the results
+
+> **P1** — *"(i) ep0's fd count falls by ≈194 within seconds … (ii) … ≈194 sockets convert ESTAB →
+> CLOSE-WAIT … (iii) neither — the count barely moves. Then the ownership attribution is wrong and the
+> finding must be withdrawn."*
+> **P2** — *"demo-hp (fixed): ≈ 0 … demo-felhom (control, untouched): ≈ 4 per hour → ≈ 16 over 4 h."*
+> **P3** — *"ep0's overall leak rate should fall from ≈ 200/day to ≈ 100/day while one box is fixed, and
+> to ≈ 0/day after Part 5."*
+
+## 5. P1 — **outcome (i)**, in one second
+
+| | ep0 fd | ESTAB | demo-felhom | demo-hp | CLOSE-WAIT |
+|---|---|---|---|---|---|
+| T-0 `09:15:46Z` | 415 | 398 | 199 | **199** | 0 |
+| T+1s `09:15:47Z` | **216** | **199** | 199 | **0** | **0** |
+| T+60s `09:16:42Z` | 218 | 201 | 199 | 2 | 0 |
+
+**Outcome (ii) did not occur, so it gets no register row.** Not one socket converted to `CLOSE-WAIT`:
+ep0 reaps on peer FIN correctly. That also means the 543 `CLOSE-WAIT` at the 2026-08-18 wedge has some
+other explanation and is **not** evidence of a second defect on the protected machine — a finding in the
+negative, worth the sixty seconds it cost.
+
+Ownership is now proven a **third** independent way: what dies with the process, agreeing with
+`ss -tnp` and with the access-log user agent. At 133 s uptime the fixed box held **0** connections.
+
+## 6. P2 — divergence
+
+**Window 09:15:47Z → 10:17:52Z = 1.03 h. You closed the ≥4 h window early**, so no daily rate is
+extrapolated and none is needed.
+
+| box | agent | start | end | delta | per hour | predicted |
+|---|---|---|---|---|---|---|
+| `demo-felhom` CONTROL | 0.129.0 | 199 | 203 | **+4** | 3.87 | ≈4 |
+| `demo-hp` FIXED | 0.130.0 | 0 | 0 | **+0** | 0.00 | ≈0 |
+
+**The assumption-free statement.** ep0's access log counts the opportunities: each box made **exactly 4
+`/snapshots` and 4 `/version` calls** in the window.
+
+> **control: 4 cycles → 4 leaks. fixed: 4 cycles → 0 leaks.**
+
+**Positive observable (standing rule 3):** a zero leak is equally consistent with "the agent stopped
+working" — it did not; its four cycles are in ep0's log. The boxes' other traffic is near-identical
+(`libwww-perl` 924 vs 926, `proxmox-backup-client` 898 vs 898), so **the only difference between them
+is the binary**. Poisson alone gives P(0 | λ=4) = **1.8%**, which is suggestive rather than conclusive
+and is not relied on alone.
+
+## 7. P3 — the second box, and the backlog clearing itself
+
+`demo-felhom` upgraded `10:18:56Z` on your word.
+
+| | ep0 fd | ESTAB | CLOSE-WAIT |
+|---|---|---|---|
+| T-0 `10:18:55Z` | 220 | 203 | 0 |
+| **T+2s** | **17** | **0** | 0 |
+| settled 10:30–10:35Z | **17–19** | 0–2 | 0 |
+
+**17 is precisely ep0's `t0` baseline** (fd 17, ESTAB 0, 2026-08-18 09:51:22Z), and it returns to 17
+between poll cycles — the "healthy proxy near 20 fds" the incident document named. Predicted ≈0/day
+residual; **observed the baseline itself.**
+
+**A correction, made within the hour it was written.** My STOP 1 report and the first CHANGELOG draft
+said *"does not clear the 388 descriptors already stuck on ep0 — those persist until that proxy
+restarts."* **Wrong.** They were held on both sides; restarting the agents released every one. **ep0 was
+read-only throughout and its proxy PID never changed (551655).** Corrected in the CHANGELOG, the audit
+document and R-344 rather than quietly edited.
+
+## 8. Fleet sanity
+
+Hub reports **0.130.0 on both** boxes. **No `floor held`** line (0.130.0 > golden MinAgent 0.129.0).
+**No `pbsdr_box_unreachable` / `offsite_box_unreachable`** during any window. Positive observable
+rather than the absent one: the PBS-DR gauge kept refreshing (`3.7% full (3.7 GB of 97.9 GB)`) and host
+reports kept landing from both boxes throughout.
+
+## 9. What is NOT done
+
+- **Not published.** No package, no tag, no manifest or floor field touched, no self-update staged.
+  **A box installed from the current image still ships the leaking agent** — **R-347**, your call.
+- The CHANGELOG heading is `## UNRELEASED — v0.130.0 candidate`. The `release-complete` gate convicted
+  on `## v0.130.0` because there is no tag and no package, and **it was right to**. I did not use
+  `--no-verify`; I made the heading stop claiming a release that has not happened. It flips to
+  `## v0.130.0` in the same commit as the tag.
+- Poll rate unchanged (**R-336**, re-scoped). `pbsTargetsFromPVE` not refactored; no
+  `CloseIdleConnections` added. ep0 not touched.
+- **The 388 descriptors ARE cleared** — see §7. This is the one item the prompt expected to remain
+  outstanding, and it did not.
+
+## 10. Register
+
+- **R-344** — updated with the fix, P1's outcome named, and P2/P3's numbers. **Left OPEN**, because a fix
+  on two boxes by hand is not delivered.
+- **R-336 — re-scoped.** Its new next-step cell, verbatim: *"**NEW ACCEPTANCE CRITERION, since the old
+  one is void:** the fd count is NOT the observable for this row any more — that belongs to R-344 and is
+  already satisfied. Measure the REQUEST RATE at ep0's access log, and state the projected rate at the
+  target customer count."* The row now records explicitly that its old next-step **would have "fixed"
+  nothing while looking like a failed fix**, and re-scopes it to what it is: ~85,000 requests/day to a
+  weekly-write DR endpoint, ≈**25 requests/second at fifty customers** against a CX33.
+- **R-347 (new)** — the delivery gap. Owner: **Viktor decides**, CC executes.
+- **R-348 (new)** — an agent restart blanks the reported backup list for up to ~18 h, and the `Store`
+  comment calls backups *"unaffected"*. **Blinds no alarm** — checked, not assumed: the hub's
+  `backupEvidenceLookback` scans 7 days for exactly this case, and `pbs_snapshots` stayed populated.
+- **No P1(ii) row**, because outcome (ii) did not occur.
+- **R-346** — this run anchored on the measured `t0` (fd 17 at 2026-08-18 09:51:22Z), never on a systemd
+  timestamp, so the 5 h 56 m discrepancy did not touch these numbers.
+
+## 11. CI, by run ID — and the claim ledger
+
+| repo | run | sha | conclusion |
+|---|---|---|---|
+| `felhom-agent` | id **362** / run_number 50 | `ede49b610` — the fix | **success** |
+| `felhom-agent` | id **363** / run_number 51 | `7569f34ae` — the correction | **success** |
+| `felhom.eu` | id **364** / run_number 239 | `57dd62b09` — the write-up | **success** |
+
+No `--no-verify` anywhere. Both repos' pre-push hooks ran their gate entry point and passed;
+`release-complete` passes on v0.129.0, which is the honest state while 0.130.0 is unpublished.
+
+`python3 scripts/unproven.py --summary` — **unchanged: 23 walked, 32 not walked of 55.** No number
+moved, and correctly so: this run proved an engineering fact about our own connection handling, not a
+customer-facing product claim.
+
+**Closing state of ep0**, read one last time after everything:
+
+```
+t=10:40:23Z pid=551655 fd=17 estab=0 ctrl(.2)=0 fix(.3)=0 CLOSE-WAIT=0
+```
+
+`CLOSE-WAIT 0`, `ESTAB` in the low single digits, `fd` at the baseline, proxy PID **551655** — the same
+process that has been running since 2026-08-18 09:51:04, never restarted by this work.
+
+## 11b. The release (R-347, CLOSED)
+
+`bash scripts/release-agent.sh 0.130.0` — the one documented way (R-115): build, tag, publish, and
+**verify by independent download**.
+
+| | |
+|---|---|
+| tag | `v0.130.0` at `7569f34` |
+| sha256 | **`a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3`** |
+| size | 14,141,158 bytes |
+| reproducible | **yes, checked** — `-trimpath -buildvcs=false` rebuild matches byte for byte |
+
+**Vouched** in the Day-0 artifact manifest: agent 0.129.0 → **0.130.0** + its sha. **Only the agent
+fields changed.**
+
+- **`min_agent` left at 0.129.0** — it states what the **golden controller** needs. Raising it to
+  0.130.0 would have made the hub **HOLD the floor** for every box below 0.130.0, which is the
+  opposite of shipping a fix.
+- **Global floor never touched** (0.216.0). On hub v0.106.0 it is a *separate form with its own
+  action*, so publish-train rule 2's "save the floor last" hazard no longer exists in the shape its
+  incident describes — the rule's reasoning holds, its mechanism has moved.
+- After: no `floor held`, no `*_unreachable`, both boxes reporting 0.130.0, artifact downloading
+  anonymously at the vouched sha.
+
+**No `--no-verify` in the train.** The heading was flipped to `## v0.130.0` only after tag and package
+existed. Flipping first and bypassing would have produced a red CI run and an alarm mail for a release
+that worked — R-168's failure mode.
+
+## 11c. Two findings from the release
+
+**R-349 — the fleet was running a different binary under the same version name.** The proof deploy was
+a hand build; the release builds `-trimpath -buildvcs=false`. Same source, same version string,
+different bytes (`256e0829…` vs `a56a92a7…`). **Self-update could never have corrected it** — the boxes
+already reported 0.130.0, so the vouched version looked installed. Every version check in the system
+compares the *string*. Fixed by installing the **downloaded** artifact on both. The proper fix already
+exists in miniature: `wrapper_sha256` does exactly this drift detection for the PBS wrapper and was
+never extended to the agent's own binary.
+
+**R-350 — I printed the hub password into the transcript.** Confirming the vouch used
+`curl -w '%{redirect_url}'`; the hub answers 303 and curl re-attaches the basic-auth credential to the
+redirect target it prints. **Not in git, not in any committed file** (checked by content), not in the
+evidence directory — it is in the session transcript on DooPlex. Every other call printed only the
+length; this came through curl's own formatting. **Rotation is your call** — I did not do it
+unilaterally, and I can do it file-to-file without printing the new value if you want. The reusable
+half: `%{redirect_url}`, `-v` and `--libcurl` all re-render a basic-auth credential.
+
+## 12. Observations
+
+- **The closure refactor is not worth doing — recommend leaving it.** With the idle timeout restored an
+  abandoned client's connection is gone in 90 s, so the standing population is bounded at about one
+  connection per box instead of growing without limit. Caching clients would add cache-invalidation
+  questions (a storage's fingerprint, token or namespace can change under it) for no observable gain.
+- **Three other `http.Transport` defaults are still missing and were left alone:** `MaxIdleConns`,
+  `TLSHandshakeTimeout` (0 = no limit; `DefaultTransport` uses 10 s) and `ExpectContinueTimeout`. None
+  accumulates, and every client bounds its request with `http.Client.Timeout`. `TLSHandshakeTimeout` is
+  the only one with a plausible failure mode — a stalled handshake over the tunnel, bounded today only
+  by the outer 30 s. Not changed, because widening the diff would have made this measurement
+  unattributable. Worth a look on its own terms; not a defect.
+- **A measurement error of mine, recorded because it nearly cost four hours.** The first P2 sampler
+  reported both per-box columns as 0 while the totals were right: `ss` prints `[::ffff:10.77.0.2]:port`
+  and my pattern expected `10.77.0.2:`. Caught 15 minutes in, because a 0/0 split cannot sum to 199.
+  Fixed, then **one sample proved by hand before committing the window** — which is what should have
+  happened first.

REPORT-drill-r95-recovery-2026-09-01.md

@@ -0,0 +1,166 @@
+# REPORT — the deletion we said is survivable: the recovery route does not exist (R-95 drill, 2026-09-01)
+
+**RUNBOOK, destructive class, `demo-hp` only. STOPPED at the end of Phase 1 on the operator's ruling,
+before any destructive step. No delete verb was issued against any live store; no byte on either
+Storage Box sub-account was written, moved or removed.** No production code, no version bump, no
+image, no golden. Evidence: `documentation/audits/evidence-drill-r95-recovery-2026-09-01/`.
+
+| # | phase | verdict | one sentence |
+|---|---|---|---|
+| 1 | snapshot reachable, and its name | **NO — and it has no reachable name** | 777,600 exact names across nine days in the vendor's own format, plus 126 alternative shapes; zero resolve, with a control proving the sweep detects a path that exists. |
+| 2 | the deletion | **NOT RUN — operator ruling** | With no recovery route, the deletion would have destroyed real history to buy nothing; put as a two-option decision, the ruling was stop. |
+| 3 | the alarm fired | **NOT RUN — and it could not have fired at the specified size** | The shipped threshold needs a fall of more than half; one app's tag is ~9 of 69. → **R-435** |
+| 4 | **the recovery** | **NOT RUN — no route exists that is not fenced** | Box-side: proven impossible. Panel: fenced and browserless. Hetzner API: fenced (§11-D), and the hub's client has no snapshot method at all. |
+| — | **RTO from T₀** | **STILL BLANK** | Row 10's RTO cell is unchanged and remains a finding. |
+| — | **data lost, quantified** | **NOT MEASURABLE THIS WAY** | The quantity only has meaning if the rest is recoverable, and the route that would recover it is not reachable. |
+| 5 | re-arm | **NOT RUN** | Depended on Phase 4. |
+| 6 | teardown | **PASS** | Store untouched at 69 snapshots; all three scratch layers removed; hub DB copies shredded; both boxes healthy. |
+
+---
+
+## 1. Did the recovery work — and does yesterday's re-scope survive?
+
+**The recovery was never reachable, and the re-scope does not survive intact. Its first half stands;
+its second half does not.**
+
+Yesterday's re-scope has two clauses. They must now be separated:
+
+* **(a) "The box can delete its live repository, but cannot write to the daily snapshots of it."**
+  **STANDS.** Re-confirmed here: `/.zfs/snapshot` is reachable and the write-refusal measurement is
+  unchanged. Nothing in this drill weakens it.
+* **(b) "…so the rest is recoverable — file by file, one customer at a time."** **NOT SUPPORTED.**
+  A snapshot that cannot be opened cannot be copied out of. R-432 recorded the directory listing
+  empty and named the cheapest next step: *"a single `ls /.zfs/snapshot/<name>` from a box then
+  settles whether a named snapshot can be entered even though the directory does not list (ZFS
+  allows exactly that)."* **That step is now done, exhaustively, and the answer is no.**
+
+**What was measured.** The port-23 restricted shell accepts a batched `stat`, which makes a cheap
+existence oracle: 500–600 paths per round trip, stdout carrying only paths that exist.
+
+| sweep | candidates | hits |
+|---|---|---|
+| `/.zfs/snapshot/YYYY-MM-DDTHH-MM-SS`, nine full days, second granularity | **777,600** | **0** |
+| 126 alternative name shapes and snapshot paths (`daily`, `snapshot-1`, colon and compact time forms, `/home/.snapshot`, …) | 126 | 0 |
+| **control — the identical 600-name batch shape with one real path appended** | 6 batches | **6/6 returned it** |
+
+**And there is a structural reason, which is why I stopped sweeping.** The customer's data and the
+snapshot door are on **different filesystems**:
+
+```
+df           →  u629488-sub3 mounted on /home
+stat /home            →  Device 0,82
+stat /.zfs/snapshot   →  Device 0,276      ← a different device
+stat /home/.zfs       →  cannot statx: No such file or directory
+```
+
+A ZFS snapshot under `/.zfs/snapshot` belongs to the dataset that owns that `.zfs` — not to the
+child mounted at `/home`. **So even a correctly named snapshot there could not contain
+`felhom-repo`,** and the dataset that does hold it exposes no `.zfs` at all to this account. The
+empty listing is not a display toggle hiding a reachable tree; from a sub-account there is no tree.
+
+**Three tools agree, each with controls in the same run:** SFTP, the port-23 shell, and
+`rsync --list-only`.
+
+**What that does to R-95.** Its *exposure* is unchanged and its *remedy* is not. Yesterday the row
+could say a deletion costs about a day because the rest comes back per-file. Today the only routes
+to "the rest" are a whole-box panel rollback (which deletes newer snapshots and hits every customer
+on the box) and the provider API (fenced, and unimplemented in the hub's client). **The re-scope's
+comfort was resting on a route nobody had walked — which is precisely the standard this project
+applies, and it is the reason this drill was called.**
+
+**The ranking is Viktor's and I am not re-ranking it.** What I will say plainly: the argument that
+moved R-95 down yesterday is the argument this drill removed. On these facts I would put it back
+where it was.
+
+## 2. The RTO
+
+**Still blank, and it stays a finding.** `07` §8 row 10's RTO cell has been empty since July and this
+drill did not fill it. Nothing was recovered, so nothing was timed. The summary line at
+`07-backup-architecture.md:948` — *"no ransomware-shaped recovery has ever been run"* — is still
+true, and is now true for a sharper reason: **not "nobody has run it" but "from the box, it cannot
+be run."**
+
+## 3. R-432's answer, and the naming scheme
+
+**R-432 is ANSWERED, and negatively. It did not need the panel read it was waiting on.**
+
+* **The naming scheme is `YYYY-MM-DDTHH-MM-SS`** — vendor-documented examples `2025-12-03T13-47-47`,
+  `2025-02-12T11-35-19`. Recorded so nobody hunts a console again.
+* **Knowing it does not help.** Every name in that format for nine days is refused, and the st_dev
+  split above says why. **Per-file recovery is not operator-only — from the box it is nobody's,** and
+  for the operator it is a browser act against the main account that no credential in this project
+  can perform.
+* **The panel cannot supply the missing piece either.** It offers Restore and Delete on a row and
+  does not show names; and the one name-shaped thing it could give would be tried against a door
+  that leads to the wrong dataset.
+
+## 4. The alarm's first real firing
+
+**It did not happen, and the drill as written could not have produced it.** The detector fires on a
+fall of **more than half** the previous count **and at least 5** (`hub/internal/monitor/offsite.go`,
+`snapshotDropFraction = 0.5`, `snapshotDropFloor = 5`). demo-hp's baseline is **69**. Phase 2 deletes
+**one app's** history — about **9** snapshots. 9 is over the floor and nowhere near half, so the
+alarm stays silent, **correctly and by design**. Firing it for real needs ~35+ snapshots destroyed,
+i.e. most of demo-hp's off-site history. That trade is what the operator was asked to rule on. → **R-435**
+
+**One thing the alarm says is now wrong.** Its message, live in hub 0.111.0, reads:
+
+> "The daily Storage Box snapshots are read-only and still hold the older copy, **so this is
+> recoverable file-by-file**; it is NOT confirmed data loss."
+
+The first clause is true; **the second promises a recovery the product cannot perform and the
+operator cannot perform without a browser and the main account.** This is this project's own
+corollary — *when a verdict changes which field it counts from, the alarm text has to change with
+it* — landing on the alarm shipped the same day. → **R-434**
+
+## 5. Findings, as register rows
+
+All four filed in `documentation/backlog/OPEN-ITEMS.md`.
+
+| row | finding |
+|---|---|
+| **R-433** | A sub-account cannot reach any Storage Box snapshot **by any name**; `/home` and `/.zfs` are different filesystems and `/home/.zfs` does not exist. Answers R-432 negatively and removes clause (b) of the R-95 re-scope. |
+| **R-434** | `emitSnapshotDrop`'s message promises file-by-file recovery that is not reachable. Live in hub 0.111.0. |
+| **R-435** | The drop detector cannot see a single-app deletion (>50% of 69 ⇒ ~35 needed). `offbox.go:1388` forgets **by tag**, so a single-tag wipe is exactly the shape the detector is blind to. Deliberate insensitivity, but the blind spot should be stated where the operator reads it. |
+| **R-436** | **LEAD, not a defect.** Hetzner's port-23 shell offers `rclone serve restic --stdio` as a server-side backend, and restic 0.14.0 recognises the `rclone:` backend (measured; control `banana:` → invalid backend; rclone is absent from the controller image). `rclone serve restic` carries `--append-only`. **This could make R-95's real prevention far cheaper than the spike concluded — no new always-on machine, no data migration.** Caveat stated up front: the **client** supplies the server command line, so a compromised box could omit the flag unless the provider pins it. Settling that is a vendor question, not a code change. |
+
+**R-432 is marked ANSWERED**; its "one panel read settles it" next step is withdrawn as unnecessary.
+
+## 6. Does `07` §8 row 10 move?
+
+**No. It stays `PARTIAL`, and its RTO stays blank.** The status was already correct for the right
+reason — *"the recovery ROUTE has never been walked, which is what PARTIAL means"* — and this drill
+found the route is not walkable from the box at all. **What the row needs is a text correction, not a
+status change:** its clause *"recoverable per-file (vendor)"* and its limit *"per-file recovery is
+operator-only today (R-432)"* both overstate what exists. Updated in place with the citation. Moving
+it only as far as the evidence goes means not moving it.
+
+## 7. What could not be tested, and why
+
+* **Whether the main account can see the snapshots.** No main-account credential exists in this
+  project — the hub holds only per-customer sub-accounts. This is the one question that would decide
+  whether per-file recovery exists *at all*, for anyone.
+* **Whether the Hetzner API can list or read a snapshot.** Fenced by the runbook (§11-D). Separately,
+  `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method** — so this route needs new code
+  regardless of the fence.
+* **The deletion, the alarm's first real firing, the recovery, the RTO, the re-arm.** Phases 2–5, not
+  run, on the operator's ruling.
+* **Whether `rclone serve restic --stdio` is pinned server-side with `--append-only`** (R-436).
+
+## 8. My own mistakes
+
+* **I ran Phase 1 before finishing Phase 0's subject-app choice, and did not say so up front.** The
+  choice depends on the snapshot's timestamp, so the order was right, but the runbook's order is the
+  runbook's and a silent reordering is the thing this project keeps getting caught by. Stated at the
+  time in the session, recorded here.
+* **My first sweep guessed the schedule instead of establishing it.** I probed 00:00 UTC and 22:00
+  UTC — 600 names — on the strength of a register line reading *"daily 00:00"*, got nothing, and only
+  then widened to whole days. The narrow sweep was worth nothing on its own: a zero over a guessed
+  window is not evidence, and I should have gone to full days first or not run it at all.
+* **I nearly reported the empty listing as "the display toggle is hiding it".** The vendor documents
+  exactly such a toggle and it fitted. The st_dev comparison — which I only ran because `df` printed
+  a filesystem name I did not expect — says the tree is on another dataset entirely. **A plausible
+  cause that fits the symptom is not a measured one**, and I had the wrong one for about ten minutes.
+* **`REPORT.md` held the only copy of the R-331 report** (hub v0.109.0, 2026-08-30) — durable content
+  living only in the overwritten file, which `CLAUDE.md:82-87` forbids. Preserved as
+  `REPORT-r331-backup-card.md` before this report replaced it.

REPORT-golden-0.216.0.md

@@ -0,0 +1,151 @@
+# REPORT — bake and vouch golden 0.216.0, closing R-334 (2026-08-18, afternoon)
+
+**Outcome: R-334 CLOSED.** Golden **0.216.0** baked, published, and **vouched by the operator**.
+`golden_currency_gate.py` is green for the first time since 2026-08-14, and `repo_gates.py` is
+**fully green — all nine gates, rc=0**.
+
+Run against the existing `documentation/runbooks/RUNBOOK-manual-build.md` §4.0 + §4.1; the run sheet
+pinned this run's numbers and the stop. Evidence:
+`documentation/tests/golden-0.216.0-2026-08-18/`.
+
+---
+
+## 1. Baselines, re-read on the machine
+
+| item | value | source |
+|---|---|---|
+| newest released controller | **v0.216.0** | `felhom-controller/CHANGELOG.md` head |
+| its floor | **`MinAgent: 0.129.0`** | second line of that header |
+| newest agent release | **v0.129.0** | `felhom-agent/CHANGELOG.md` head |
+| newest golden before this run | **0.214.0** | `documentation/tests/golden-0.214.0-2026-08-12` |
+
+**All four match the run sheet's §1 — no disagreement to report.** All three repos were clean with
+`HEAD == origin/main` before starting.
+
+## 2. The published agent artifact exists
+
+Checked against the **package registry**, not inferred from a CHANGELOG:
+`generic felhom-agent 0.129.0` is published. Vouching `agent_version` at a version that was never
+published would point day-0 installs at a 404.
+
+**The R-216 check passed on the machine rather than on the coincidence.** `MinAgent` (0.129.0) is
+**equal to**, not above, the newest published agent (0.129.0). Had it read higher, hub v0.97.0 would
+hold the fleet against a version nobody has.
+
+## 3. Identity, and a verification beyond what was asked
+
+```
+GOLDEN_VERSION = 0.216.0
+GOLDEN_SHA256  = ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b
+URL            = https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.216.0/golden.tar.zst
+archive        = 656,970,239 bytes (rootfs 32G + ONE data volume 24G @ /var/lib/felhom)
+controller     = gitea.dooplex.hu/admin/felhom-controller:0.216.0
+template       = debian-13-standard_13.6-1_amd64.tar.zst  (listed live per §4.1 step 2, not reused)
+```
+
+The URL resolves (HTTP 206 on a range request). **I did not stop at the script's printed hash**: the
+artifact was downloaded back out of Gitea and hashed, and it matches `GOLDEN_SHA256` exactly. The
+script reporting a digest and the registry serving those bytes are two different claims, and only the
+second one is what a new install actually receives.
+
+## 4. Pass markers — the corrected list, quoted from the real log
+
+```
+ 82 :   docker OK (overlay2; data-root /var/lib/docker)
+313 : INFO: including mount point rootfs ('/') in backup
+314 : INFO: including mount point mp0 ('/var/lib/felhom') in backup
+319 : [golden]   pre-delete existing: HTTP 404 (404/204 expected)
+320 : [golden]   upload OK (HTTP 201)
+```
+
+`excluding` and `FATAL`: **absent**. There is no mp1 — R-165 collapsed the two data volumes into one,
+which is exactly why the pre-2026-08-06 marker list could never match and why R-233 rewrote it.
+
+## 5. Token handling, and why the control is not ceremony
+
+Copied **file → file** by `scp`; the runner script inside the VM read it from `/root/.gitea-token`
+itself, so the value never reached a command line or a unit's properties:
+
+```
+systemctl show golden-bake -p Environment -p ExecStart | grep -c -F "<token>"   =  0
+```
+
+Token-leak grep on the **committed** log, positive control run **first**:
+
+```
+seeded throwaway copy  = 1     ← proves the grep can see a token when one is present
+committed bake.log     = 0     ← the real measurement, now worth believing
+```
+
+**A `grep -c` that matches nothing returns `0`, which is indistinguishable from a clean file.**
+Without the control, the `0` is an assumption wearing a number's clothes. Both figures are from the
+copy that is committed to the repository, not only the one inside the VM.
+
+## 6. Teardown
+
+`pct destroy 9100 --purge` (both LVs removed, CT purged) → `shred -u` on the token, runner,
+build script and log **after** the log was copied out (standing rule 5) → all four confirmed absent
+→ `poweroff` → waited for qemu to exit using `ps -eo comm` (**not** `pgrep -f`, which self-matches and
+reports a false "still running") → `qemu-img snapshot -a virgin`, disk reverted, snapshot list shows
+the single `virgin` entry.
+
+**Nothing was provisioned that outlives this run.**
+
+## 7. The vouch, and its verification
+
+**Performed by the operator (Viktor)** in the hub, Configuration → Day-0 artifacts. Verified
+afterwards by reading the hub's own store rather than trusting the save:
+
+| field | value | recorded |
+|---|---|---|
+| `artifact_golden_version` | **0.216.0** | 2026-08-18 11:00:59 |
+| `artifact_agent_version` | **0.129.0** | 2026-08-18 11:00:59 |
+| `artifact_min_agent` | **0.129.0** | 2026-08-18 11:01:00 |
+| `artifact_golden_sha256` | `ac004dc9…c34b` | 2026-08-18 11:01:00 |
+
+The recorded sha256 **matches the artifact I downloaded and hashed independently** — so the hub is
+vouching the bytes that are actually published, not merely a matching version string.
+
+**This separate check was necessary, and the gate says so itself.** `golden_currency_gate.py`'s own
+pass line reads *"this checks the BAKE, not the vouch"*. A green gate on an unvouched bake is exactly
+the "baked-but-unvouched golden is worse than none" state R-334 warned about, so the gate alone could
+not have closed this row.
+
+## 8. Gates
+
+```
+golden_currency_gate.py                        rc=0
+  newest released controller : 0.216.0
+  newest golden baked        : 0.216.0
+
+repo_gates.py --fast                           rc=0
+  site OK · hostinstall OK · hub-confirm OK · manifest-bearer OK · reuse-refs OK
+  instructions OK · golden-currency OK · wire-contract OK · hub-copy OK
+  all felhom.eu gates OK
+```
+
+**This is the first fully green gate run since 2026-08-14**, and it is the point of the run: the
+CI failure mail that has been arriving since then should now stop.
+
+## 9. Documentation not changed, deliberately
+
+**`documentation/architecture/00-capability-map.md` — no change, and the reason matters.** The run
+sheet said to update it *if the day-0 install row's evidence citation names the golden version*. It
+does not: that row cites `DRILL-day0-vm-2026-07-12` / `DRILL-day0-take2-2026-07-12`. The only golden
+version literal in the map is `tests/golden-0.205.0-2026-08-07` on the **recovery-journey** row,
+which is a **dated historical citation** of what a fresh install landed on during the 2026-08-07
+walk. Bumping it to 0.216.0 would falsify a record of what happened on a specific date — `docs.md`
+permits historical citations precisely because they cannot go stale.
+
+## 10. Observations, not acted on
+
+- **`min_controller_version` in the hub still reads `0.214.0`** (last touched 2026-08-12). That is a
+  different field from the three vouched here — it is the floor the fleet is held to, not the day-0
+  golden — and it was outside this run's scope. But it is now two releases behind the golden a new
+  box receives, and STATUS.md's "approved pair" line describes it. Worth a decision; **not** changed
+  here, because widening scope past the three named fields is how a vouch goes wrong.
+- **`pveam available` still offers `debian-13-standard_13.6-1_amd64.tar.zst`** — the same point
+  release the runbook recorded on 2026-07-31. Listed live rather than assumed, per §4.1 step 2; the
+  instruction stands even when the answer happens not to have moved.
+- **The bake ran in ~5 minutes** (12:49 launch → 12:54:14 archive), well inside the drill VM's normal
+  envelope; no timeout or retry was needed.

REPORT-hub-blindness.md

@@ -0,0 +1,234 @@
+# REPORT — the hub says something when it loses sight of the off-site endpoints (2026-08-18)
+
+**Shipped: hub v0.106.0, deployed and verified.** Both box checkers now carry a second, independent
+**reachability** signal with paired all-clears. The fill logic is untouched. **Part 6 was done, not
+dropped.**
+
+**NOT proven live** — see §7. No real or constructed outage has exercised the emit path.
+
+---
+
+## 1. Confirmed baselines
+
+| item | value |
+|---|---|
+| felhom.eu `main` @ start | `78a244bf0f…` — **matches the prompt's anchor** |
+| hub version in → out | **v0.105.0 → v0.106.0** (read from `hub/CHANGELOG.md` head) |
+| `scripts/` version in → out | `due_checks_gate.py v1.0.0` → **v1.0.1** |
+| highest R in use at start | R-343, so **R-339 / R-340 free** as specified |
+
+**Had the three target files moved?** No. Verified by hash before editing:
+
+```
+61a16462756099d2fa60dd0c50aeac8c  internal/monitor/pbsdr_box.go
+d12b3e665d729a0291d2397895f23d1f  internal/monitor/offsite_box.go
+702d17487ffb4ac17a9d18050bb546b7  internal/notify/dispatcher.go
+```
+
+All landmarks in §5 of the prompt resolved as described; nothing was stale.
+
+## 2. Files created / modified
+
+**Created:** `hub/internal/monitor/box_reachability_test.go`,
+`hub/internal/notify/dispatcher_box_reachability_test.go`, `REPORT-hub-blindness.md`.
+
+**Modified:** `hub/internal/monitor/pbsdr_box.go`, `hub/internal/monitor/offsite_box.go`,
+`hub/internal/notify/dispatcher.go`, `hub/cmd/hub/main.go`, `hub/CHANGELOG.md`,
+`hub/internal/monitor/{pbsdr_box_test.go,offsite_box_test.go}` (new constructor arg),
+`manifests/hub.yaml`, `CONTEXT.md`, `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md`,
+`documentation/architecture/00-capability-map.md`, `scripts/due_checks_gate.py`,
+`scripts/instructions_gate.py`, `scripts/test_due_checks_gate.py`,
+`scripts/test_instructions_gate.py`, `scripts/CHANGELOG.md`.
+
+## 3. Commits pushed to `main`
+
+| commit | contents |
+|---|---|
+| `ab2262c91c1e976ae3b983e9df6b45dabfcb9d23` | the code, tests, and register/doc edits |
+| `c03f629d43ed3d175e2c8561486cadc9a432640f` | `manifests/hub.yaml` 0.105.0 → 0.106.0 — the change that actually deploys |
+| *(Part 6 commit — see §10)* | the today-override announcement + `scripts/CHANGELOG.md` |
+
+## 4. Tests and the three red-proofs
+
+**All named tests pass.** Groups A–F in `internal/monitor/box_reachability_test.go`, Group G in
+`internal/notify/dispatcher_box_reachability_test.go`:
+
+| test | result |
+|---|---|
+| `TestPBSDRBox_Unreachable_SustainedOutage` (A) | PASS |
+| `TestPBSDRBox_Unreachable_BlipBelowThreshold` (B) | PASS |
+| `TestPBSDRBox_Unreachable_Recovery` (C) | PASS |
+| `TestPBSDRBox_UsageUnsupported_IsNotBlindness` (D) | PASS |
+| `TestPBSDRBox_BornBlind_StillReports` (E) | PASS |
+| `TestOffsiteBox_Unreachable_AndRecovery` (F) | PASS |
+| `TestPBSDRBox_ZeroCapacitySuccess_ClearsBlindness` (§8 truth table) | PASS |
+| `TestBoxRecovery_ReachesTheOperatorDespiteInfoSeverity` (G) | PASS |
+| `TestBoxUnreachable_ReachesTheOperatorOnItsOwnSeverity` (G) | PASS |
+| `TestBoxRecovery_PairedWithTheCorrectDownType` (G) | PASS |
+
+**None of the three red-proofs passed on the first attempt** — each turned its test red, and each did
+so **for the reason under test**, which I checked in the message rather than in the count.
+
+**Red-proof 1 — threshold 3 → 1.** Group B seen failing:
+
+> `box_reachability_test.go:132: two failed windows emitted [pbsdr_box_unreachable pbsdr_box_unreachable], want silence below the threshold`
+
+The message names the premature events, not an incidental error. **Reverted** (`defaultBoxUnreachableWindows = 3` restored).
+
+**Red-proof 2 — remove the `ErrUsageUnsupported` counter guard** (deleted its early return so the
+branch falls through). Group D seen failing:
+
+> `box_reachability_test.go:213: ErrUsageUnsupported emitted [pbsdr_box_unreachable × 8] — an expected pre-update condition must never alert`
+
+The message names the **unexpected event type**, as the prompt required — not merely a count.
+**Reverted.**
+
+**Red-proof 3 — remove `pbsdr_box_recovered` from `recoveredPairedDownTypes`.** Group G seen failing,
+and the first failure is the **end-to-end mail assertion**, which is what proves the test exercises
+the wiring rather than the map:
+
+> `dispatcher_box_reachability_test.go:41: pbsdr_box_recovered: operator mails = 0, want 1 — the all-clear must reach the operator; 0 means the recoveredPairedDownTypes entry is missing and "info" was dropped by the severity gate`
+
+**Reverted.** `grep -rn MUTATED internal/` returns nothing.
+
+## 5. Test count
+
+`go test ./...` — **21 packages, all green** (18 with tests, 3 with none). `internal/monitor` gained 7
+tests; `internal/notify` gained 3. `go build ./...` and `go vet ./...` clean.
+
+Repo gates: **10/10 OK, rc=0**.
+
+## 6. Deployed version and the wiring evidence
+
+```
+ArgoCD app "felhom":  Synced  Healthy   rev=c03f629d43ed3d175e2c8561486cadc9a432640f
+pod:                  hub-654bbc8fbc-9wld9   1/1 Running
+running image:        gitea.dooplex.hu/admin/felhom-hub:0.106.0
+```
+
+**The required post-deploy check — both constructor log lines carrying the threshold:**
+
+```
+19:29:07 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%,
+                oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
+19:29:07 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%,
+                unreachable after 3 consecutive failed reads, refresh 15m0s
+```
+
+**Two lines, both carrying the threshold — the parameter reached both checkers.** Their absence would
+have meant the config was inert however green the tests were. Note this also exercised the
+**absent-key** path: `box_unreachable_windows` is deliberately not in any deployed config, so both
+checkers fell back to the documented default of 3, which is what the log shows.
+
+## 7. NOT yet live-validated — explicitly
+
+**No real or constructed endpoint outage has exercised the emit path end to end.** Everything in §4
+is an injected fake with a scripted error and an injected clock. What is proven: the checkers emit the
+right events with the right details, and the dispatcher routes both new `*_recovered` types to a real
+operator mail. What is **not** proven: that a genuine ep0 or Hetzner failure produces those errors in
+the shape the checkers expect.
+
+A real outage cannot be manufactured without making ep0 or the Hetzner API unreachable, and **ep0 is
+Tier 2 protected — that was not done.** The constructed-outage option, named but not performed: point
+the tenantsync client at a blackholed address on a **scratch** hub instance and let three windows
+elapse.
+
+## 8. Teardown
+
+**This run provisioned nothing.** No VM, no container beyond the hub's own rolling deployment, no
+drill target, no scratch guest. All three layers N/A. `ep0`, both demo boxes and the drill VM were
+untouched, as were the agent's credential-consume and self-heal paths.
+
+## 9. Register rows
+
+**R-339 opened and marked SHIPPED** (hub v0.106.0), with PROVEN-LIVE explicitly still owed and an
+instruction not to close it on the unit tests.
+
+**R-340 opened, READY (M)** — the honest boundary: the hub's ep0 read is the `usage` op, which rides
+the **local API daemon**, and the 2026-08-18 incident explicitly cleared that daemon while the HTTPS
+proxy on 8007 was wedged. **R-339's check would have shown green for all 9 h 37 m of the outage that
+motivated it.** Overlap with R-336's remaining half is noted so whichever runs second reuses the
+first's evidence rather than re-measuring a protected machine.
+
+**R-336's next-step cell corrected. The replacement text, verbatim:**
+
+> **CORRECTED 2026-08-18 (evening) — the easy lever named here does not exist.** This cell used to
+> read *"PVE storage status is the prime suspect, and its interval is tunable"*. **The first half is
+> right and the second half is false.** `pvestatd` stats EVERY configured storage on each 10-second
+> cycle, and Proxmox staff have stated the interval is not designed to be configurable — so there is
+> no knob to turn down. The only lever PVE actually offers is disabling the storage entry
+> (`pvesm set <id> --disable 1`) around the backup window, and that is **substantially more than a
+> tuning knob**: it collides with `felhom-agent/internal/pbsdr/manager.go`'s health model, where an
+> inactive-but-existing entry drives the consume-the-one-time-secret recovery path. So the fix is a
+> design question (does the hub still need a 15-minute fill reading at all, given R-339 now reports
+> reachability separately?), not a config edit. **Doc-only correction — no agent code was changed.**
+> The remaining step is unchanged: cut the poll rate by whatever means survives that question, then
+> confirm the fd count between restarts stops climbing — the positive observable, per standing rule 3.
+
+## 10. Part 6 — DONE, not dropped
+
+Both gates now announce the `FELHOM_GATE_TODAY` override loudly before any verdict, and
+**`instructions_gate.py` no longer swallows a malformed one** — it used to fall through to the real
+date in silence while `due_checks_gate.py` already exited 2 on the same input, so one variable had two
+gates disagreeing about what a mistake means. Both exit 2 now.
+
+Tests extended in both suites (**42** and **73** assertions, all green). Red-proof: the announcement
+was deleted from `due_checks_gate.py` and its two assertions were seen failing —
+`P6: valid override is announced` and `P6: the announcement says the real date is being ignored` —
+then reverted.
+
+## 11. Gate and CI status
+
+`python3 scripts/repo_gates.py` → **rc=0, all ten gates OK**, including `due-checks`.
+
+**The due-checks gate did NOT refuse this push.** R-341's first check comes due 2026-08-19 UTC and
+this work ran on 2026-08-18 (17:18–19:30 UTC), so the gate reported *"2 dated check(s) pending, none
+due yet"* throughout. **No row was cleared, no date edited, no `--no-verify` used.** Every push today
+went through the armed hook.
+
+**CI, by run ID — all three pushes green:**
+
+| run | head_sha | conclusion |
+|---|---|---|
+| **357** | `ab2262c91` | success — the code, tests and register edits |
+| **358** | `c03f629d4` | success — the manifest bump that deployed it |
+| **359** | `104ef34f5` | success — Part 6 and this report |
+
+(Run 356 on `78a244bf0`, the baseline, was also green — so these greens are attributable to this
+work rather than inherited from a red baseline. Per §13 of the prompt, a red run here would have been
+mine.)
+
+## 12. `unproven.py --summary`
+
+```
+where felhom stands — 55 claims, verified_on 2026-08-09
+  walked   23
+  partial  14   (6 cite evidence, 8 prose only)
+  built    14   (0 cite evidence, 14 prose only)
+  missing  4   (0 cite evidence, 4 prose only)
+  NOT WALKED: 32 of 55
+```
+
+**No number moved.** Correct: this shipped an implemented-not-proven capability, which is exactly the
+status that does not advance the walked count. Moving it would require the live validation §7 says
+has not happened.
+
+## 13. Observations — noticed, deliberately not acted on
+
+- **`make docker-push` also tags and pushes `:latest`**, which the project's own rules forbid. I used
+  `make docker` followed by an explicit `docker push …:0.106.0` instead, so no `:latest` was moved.
+  The Makefile target is a loaded gun for anyone who runs the documented command; not changed here
+  because it is outside this task's scope.
+- **`internal/monitor/storage_fill_test.go` is not gofmt-clean, and was already so at `HEAD`** —
+  confirmed by stashing my changes and re-running `gofmt -l`. Not touched; it is not mine and fixing
+  it would put unrelated churn in this diff.
+- **The two checkers are now ~95% identical in their reachability half.** A shared helper is the
+  obvious next move and was deliberately not done here, per the prompt: they have different sources,
+  different error taxonomies (one has a sentinel, one does not) and different snapshot types, and the
+  existing code keeps them separate on purpose. Worth revisiting if a third box checker appears.
+- **The first ArgoCD sync reported `Synced/Healthy` at the PREVIOUS revision** (`ab2262c`) while the
+  pod was still `ContainerCreating`. Waiting and re-reading gave `c03f629` and the correct image. A
+  sync status sampled too early is not the deploy's verdict — the running image tag is.
+- **`alerting.box_unreachable_windows` is in no deployed config file**, by design, so the live hub is
+  running on the compiled default. If the operator wants to tune it, the key has to be added to the
+  hub ConfigMap first.

REPORT-r331-backup-card.md

@@ -0,0 +1,187 @@
+# REPORT — R-331: the operator Backup card said every customer had no backups
+
+**Hub v0.109.0 (with controller v0.225.0) · 2026-08-30**
+
+---
+
+## 1. What was wrong
+
+The hub customer page's **Backup** card read, for **every customer, indefinitely**:
+
+```
+Enabled  Yes        Snapshots  0
+Repo Size  0 MB     Integrity  Unknown
+```
+
+Measured on `demo-hp` 2026-08-30, at which moment the truth was:
+
+| source | value |
+|---|---|
+| the box's own `settings.json` | `snapshot_count: 67, repo_size_bytes: 140829678, stats_known: true` |
+| that night's controller log | `[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s` |
+| **this hub's own Offsite page** | `0.1 GB` used of a `50 GB` quota — read from the same stored report |
+
+**A card that reads "no backups" over a working backup is worse than no card.** It is the R-88
+direction of failure — degrading to *no backup* rather than to *unknown* — on the one screen an
+operator consults to answer "is this customer protected?".
+
+## 2. Root cause
+
+The card rendered the report's **`backup`** object. Its `snapshot_count`, `repo_size_mb` and
+`integrity_ok` fields have had **no producer** since disk-tier restic moved to the host agent (slice
+8C) — the controller's `buildBackupReport` leaves them zero *deliberately* and says so in a comment.
+The zeros were correct values for dead fields, rendered as if live.
+
+**The data was never missing.** The live numbers ride in the report's **`offsite`** object, which this
+package **already** reads for the Offsite page (`offsiteUsageBytes`) and which `monitor.OffsiteChecker`
+**already** drives fill and staleness alarms from. That the Offsite page rendered demo-hp's real usage
+from the same stored report, at the same moment the Backup card said `0 MB`, is the proof the bytes
+were arriving. This is a **render fix over an existing feed**, not a new pipeline.
+
+## 3. Why it was not a one-line template swap
+
+`snapshot_count: 0` means two opposite things — *this repository holds nothing* and *nobody has ever
+measured this repository*. **R-225 measured that confusion one layer down**: a rebuilt box rendered
+„Tarolo meret · 0 pillanatkep" over a store that really held snapshot `f3d9cd67`, and the controller's
+`StatsKnown` fixed it there. It was never on the wire, so rendering the count without it would have
+**moved R-225 up to the hub instead of fixing anything**. Controller v0.225.0 now forwards
+`stats_known`.
+
+## 4. What changed
+
+`hub/internal/web/backup_card.go` builds a typed `backupCardView` — resolved in Go, because the card's
+whole subject is a distinction a template `{{if}}` chain over `map[string]interface{}` float64s cannot
+keep:
+
+| report state | card shows |
+|---|---|
+| no `offsite` object at all | "No off-site data reported" — **and says explicitly this is not the same as "no backups"** |
+| `enabled:false` + declared `state` | the blocker by name (`needs_credential`) — a different operator action from "not enabled" |
+| enabled, `stats_known:false` | **—**, plus "never been measured". Never `0` |
+| enabled, `stats_known:true` | the real count and size, **including a real `0`** — measured empty is knowledge |
+
+**A pre-v0.225.0 controller sends no `stats_known`, which unmarshals to false → "unknown".** That is
+the fail-safe direction: upgrading the hub ahead of the fleet must not tell the operator that every
+un-upgraded customer has zero backups. Pinned by a test.
+
+**The Integrity row is deleted, not re-sourced.** Nothing produces it: the controller runs no integrity
+check, and `NotifyIntegrityOK` / `NotifyIntegrityFailed` exist and are **called from nowhere**. A row
+that can only ever read "Unknown" is not information, and one that could read "OK" from an unwritten
+field would be a lie.
+
+`fmtBytesAuto` is new rather than reusing `fmtBytesGB`: that one is fixed at GB because it renders
+against GB quotas, and it turns demo-hp's real 140 829 678 bytes into `0.1 GB` — which on a card whose
+entire defect was under-reporting a real backup reads as "nearly nothing".
+
+## 5. Tests and the red-proof
+
+`r331_backup_card_test.go` asserts the **rendered page**, using demo-hp's real reported values, so a
+regression fails against the same numbers the defect was measured against. **The defect lived in the
+template's choice of source object, so a test one layer below it would have been green against the
+shipped bug** — which is why these drive `handleCustomerUnified` and grep the HTML.
+
+**RED-PROOF (run 2026-08-30):** restoring the pre-fix card markup fails all four tests —
+`the rendered Backup card does not contain demo-hp's real snapshot count (67)`,
+`the card does not carry the real repository size (134.3 MB ...)`,
+`the card still shows an Integrity row`, plus every branch of the three-way ruling. Restored
+immediately; `git diff` clean.
+
+**Green gate:** `go build ./... && go vet ./... && go test ./...` in `hub/` — 18 packages, rc 0.
+
+## 6. Deployment and live verification
+
+Hub **0.109.0** built, pushed, manifest bumped, ArgoCD hard-refreshed and synced. Controller
+**0.225.0** deployed to both demo boxes. Verified from the live objects, not from a rollout message
+(an ArgoCD "rolled out" can name the old image):
+
+```
+argocd:  sync=Synced  rev=36f86300209e9ce914f2a97dac16ee9eeea249e2   (== HEAD)
+deploy:  gitea.dooplex.hu/admin/felhom-hub:0.109.0
+pod:     hub-6795879c4b-pf4lz  ...felhom-hub:0.109.0  Running
+boxes:   ...felhom-controller:0.225.0  Up (healthy)   on demo-felhom AND demo-hp
+```
+
+**The card, fetched from the live hub** (endpoint-level: the exact URL the operator's browser
+requests; the residual is client-side rendering only — there is no browser on DooPlex):
+
+| | demo-hp | demo-felhom |
+|---|---|---|
+| Off-site snapshots | **67** | **10** |
+| Repo size | **134.3 MB** | **132.5 KB** |
+| Last successful run | 15h ago | 15h ago |
+| Soft quota | 50 GB | 50 GB |
+| Integrity row | **absent** (grep count 0) | **absent** (grep count 0) |
+
+Both read `Snapshots 0 · Repo Size 0 MB · Integrity Unknown` before this change.
+
+**Cross-checked against the source, not just against itself** — the numbers on the card are the
+numbers on the boxes:
+
+```
+demo-hp      settings.json offbox: snapshot_count 67, repo_size_bytes 140829678, stats_known true
+demo-felhom  settings.json offbox: snapshot_count 10, repo_size_bytes 135635,    stats_known true
+                                   135635 / 1024 = 132.5 KB  → matches the rendered value
+```
+
+**One honest detail worth keeping:** demo-felhom's "Last DB dump" reads `—`. That is correct, not a
+regression — its only app (`opengist`) has no database, so the box has never taken a DB dump.
+
+**Not verified live: the "never measured" branch.** Both boxes report `stats_known: true`, so the
+degradation path could not be exercised on real hardware without falsifying a box's state. It is
+covered by `TestBackupCard_ThreeWayRuling` and `TestBackupCard_OldControllerDegradesToUnknownNotEmpty`
+at render level, and this is stated rather than implied.
+
+## 7. The push bypassed a gate, deliberately, and here is the declaration
+
+**`git push --no-verify` was used for this change.** `repo_gates.py`'s `golden-currency` gate was
+CONVICTED and it was RIGHT: controller **v0.224.0** and **v0.225.0** are released and the newest golden
+bake carries **0.223.0**, so a machine installed right now receives neither fix.
+
+**This is a BYPASS, not a waiver.** The gate offers a waiver only for a release that *deliberately needs
+no golden*; these need one. **The operator was asked and ruled bypass-now-bake-later**, on the stated
+ground that neither fix bites a day-0 box — R-330 is a nightly false alarm about apps a new box has not
+installed yet, R-331 is a hub-side display over backups a new box has not taken yet — and both arrive by
+self-update afterwards. That ground is recorded on R-242 precisely because it is the thing to re-check:
+**it does not extend to a release that changes first-boot behaviour.**
+
+**A golden carrying 0.225.0 is OWED** (`RUNBOOK-manual-build.md` §4.1; the vouch is a three-field change,
+`MinAgent 0.129.0`). This is the **fourth** bypass of this gate, and the gap it names is now two releases
+wide rather than one.
+
+**The other failing gate was fixed, not bypassed.** `due-checks` was red on R-341's `+7 d` measurement,
+five days overdue. It was **taken** during this session — see §8.
+
+## 8. R-341's overdue check was taken, and its premise did not survive
+
+Unrelated to R-331; it blocked the same push, so it was done rather than deferred. Evidence:
+`documentation/audits/evidence-r341-plus7d-2026-08-30/step1-fd-and-sockets.txt`.
+
+**Precondition passed**, which is what makes the reading interpretable: `proxmox-backup-proxy` still
+`MainPID 551655`, `ps -o lstart=` still `2026-08-18 09:51:04`, `NRestarts=0` — the same proxy generation
+as t0, so nothing restarted and re-based the count. (The anchor is `ps`, not `ActiveEnterTimestamp`,
+which reads 03:54:54Z here — R-346's trap, avoided.)
+
+**Result: fd = 17.** Not 17 more — seventeen total, exactly the documented baseline, against **405** at
+the first check on 2026-08-20. The socket histogram holds **one LISTEN and nothing else**: ESTAB 0,
+CLOSE-WAIT 0.
+
+**The verdict is "unanswerable", not "the upgrade fixed it".** R-341 asks whether the PBS 4.2.5-1
+upgrade changed the fd slope. Inside this interval **we removed the leak ourselves** (R-344, agent
+0.130.0, now live on both boxes). A slope of ~0 measures our fix, not the upgrade — and reading it the
+other way would credit a changelog that was read in advance and found to contain no such mechanism. The
+perturbation pre-registered for this window was Phase C at ~3%; the actual perturbation was the removal
+of the entire phenomenon. The question is now **moot**, and the row is closed as such.
+
+**What it does establish, which is worth more than the original question:** twelve days after the R-344
+fix, on the same proxy generation with no restart to hide behind, ep0 sits at baseline with zero
+established connections. The 388-descriptor accumulation has not returned, and R-336's ~323-day runway
+concern retires with it.
+
+## 9. Not done, and why
+
+- **No staleness verdict on the card.** `monitor.OffsiteChecker` already owns that and alarms on it. A
+  second verdict over the same data is two things that can disagree — a shape this codebase has already
+  paid for (`LastRun` vs `LastSuccess`, R-100).
+- **The dead `backup` fields were not removed from the controller's wire format.** Removing them would
+  stop historical reports already in this hub's store from parsing, for no gain — nothing renders them
+  now, and a controller-side test fails if anything starts producing them.

REPORT-r431-snapshots.md

@@ -0,0 +1,209 @@
+# REPORT — R-429 corrected, R-95 re-scoped, R-431 shipped (2026-09-01)
+
+## Part 1's answer, first, because everything reads differently after it
+
+**Can a customer's own account reach the snapshot tree? NO — it can reach the DOOR, and is refused
+writes to it, but the tree lists EMPTY.**
+
+Measured on **both** live boxes, over the credential each already holds, with controls in the same run.
+
+```
+=== POSITIVE CONTROL: the account home (must list) ===
+    drwxr-xr-x  <user> 1058   4 Aug 22 03:10 ./.
+    drwx------  <user> 1058   3 Jul 23 09:53 ./.ssh
+    drwxrwxr-x  <user> 1058   8 Aug  4 12:38 ./<repo>
+=== NEGATIVE CONTROL: a name that cannot exist (must fail) ===
+    Can't ls: "/home/./zzz-no-such-r429" not found
+=== CANDIDATE A: /.zfs/snapshot ===
+    drwxrwxrwx  root root   2 Jan  1  1970 /.zfs/snapshot/.
+    drwxrwxrwx  root root   0 Jul 21 16:01 /.zfs/snapshot/..
+=== CANDIDATE B: /home/.zfs/snapshot ===
+    Can't ls: "/home/.zfs/snapshot" not found
+=== CANDIDATE D: is the .zfs door itself visible? ===
+    drwxrwxrwx  root root   2 Sep  1 12:12 /.zfs/shares
+    drwxrwxrwx  root root   2 Jan  1  1970 /.zfs/snapshot
+```
+
+**The write-refusal test — the load-bearing sentence of the whole re-scope, now proven not cited:**
+
+```
+--- control: the same write in the account home MUST succeed
+    sftp> put … ./r429-write-control.txt
+    Uploading … to /home/./r429-write-control.txt
+    -rw-r--r--  <user> 1058  11 Sep  1 12:12 ./r429-write-control.txt
+--- cleanup of the control file
+    Removing /home/./r429-write-control.txt
+    Can't ls: "/home/./r429-write-control.txt" not found
+--- THE TEST: write into /.zfs/snapshot (must be REFUSED)
+    Uploading … to /.zfs/snapshot/r429-write-attempt.txt
+    dest open "/.zfs/snapshot/r429-write-attempt.txt": Failure
+--- confirm nothing was left behind:
+    drwxrwxrwx  root root  2 Jan  1  1970 /.zfs/snapshot/.
+```
+
+**Identical on demo-felhom** (gid 1019). `storage-box-pool-1` **is** `u629488`
+(`RUNBOOK-ep0-datastore-volume-2026-07-27.md:386`) — the same box that demonstrably holds seven
+snapshots — so the emptiness is **per-sub-account filtering**, not absence. → **R-432**.
+
+## What changed in the record
+
+| where | from | to |
+|---|---|---|
+| **R-429** | *"the mitigation has never been confirmed"* | **CLOSED — confirmed working.** My probe used `.snapshots`; the vendor documents `/.zfs/snapshot`. The controls were sound, the subject was wrong. What remains true is the actual finding: the row had no id, its "confirm tomorrow" went 36 days unanswered, and `DUE-CHECKS` was empty. **The finding was never the snapshots — it was that nobody could tell.** |
+| **R-95** | *"can delete, exposure open-ended"* — #1 since July | **RE-SCOPED:** deletes the live repo, **cannot write to the snapshots of it**; costs ≤1 day plus per-file recovery. **Ranking left to Viktor.** |
+| **07 §8 row 10** | *"the restic repo is NOT protected the same way"* | **both halves stated:** (a) the live repository is deletable — R-95 stands; (b) the snapshots are not writable by anything — proven. **Status NOT moved** — the recovery route has never been walked, which is what PARTIAL means. |
+| **07 §10.2** | R-95's line | gains the re-scope + citation |
+| **§11-D** | — | **untouched.** Nothing here answers whether the two Hetzner services share an account. |
+
+## Part 3 — what normal looks like, in numbers
+
+Hub's own `reports` table: **12 898 reports, 4 customers, 2026-06-05 → 2026-09-01.**
+
+- **Nine decreases in the entire history, and every one lands exactly on ZERO** — 36→0, 18→0 ×2,
+  15→0, 12→0, 8→0, 3→0. **Not one gradual retention decrease anywhere.**
+- **Every one predates `stats_known`** — the R-331 shape, a zero meaning *unmeasured*. Several carry a
+  declared `State` (`needs_credential`, `awaiting_recovery_key`) saying so outright.
+- **In the `stats_known`-true window (380 reports) there are ZERO decreases**: demo-felhom flat at 10;
+  demo-hp 67→68→69, rises only.
+
+**So observed churn gave nothing to calibrate against, and I say so rather than inventing a number.**
+
+## The threshold, and where it came from
+
+**A fall of more than HALF the previous count, AND at least 5.** Reasoned from what retention *can*
+do, since it was never seen to do anything: `--keep-daily 7 --keep-weekly 4 --keep-monthly 6
+--group-by host,tags` over ~8 apps **cannot halve a total** — those floors are per group — while a
+mass deletion goes to ~0. The floor of 5 stops a small-count box twitching. Deliberately not
+sensitive: a detector that cries wolf is switched off within a fortnight.
+
+## Files, commits, deployment
+
+| file | what |
+|---|---|
+| `hub/internal/monitor/offsite.go` | +170 lines: third signal, three guards, escalation latch |
+| `hub/internal/monitor/offsite_r431_test.go` | NEW — 5 tests |
+| `hub/internal/monitor/testdata/r431_real_history.json` | NEW — 9 009 real points, committed |
+| `hub/internal/api/handler.go`, `hub/internal/notify/dispatcher.go` | allowlist + operator-only, same commit |
+| `hub/CHANGELOG.md`, `CONTEXT.md`, `STATUS.md`, `07`, `00-capability-map.md`, both registers | the record |
+
+Commits: **`3068176`** (code + record), **`65c82c4`** (manifest bump).
+**Deployed hub `v0.111.0`.** ArgoCD: `sync=Synced health=Healthy`,
+`rev=65c82c4aa0949510670c42eb5c25515c25e12040` — **exactly HEAD**; `deployment "hub" successfully
+rolled out`; live image `gitea.dooplex.hu/admin/felhom-hub:0.111.0`; startup line
+`Offsite checker initialized: … 3 ok-seeded`. Image presence verified in the registry before syncing.
+
+## Tests and red-proofs
+
+| test | result |
+|---|---|
+| `TestR431_FiresOnAMassDeletion` | 69→4 fires exactly once, severity in the vocabulary, message carries both numbers and does not claim loss |
+| `TestR431_SilentWhenNotTrustworthy` | silent on all four shapes (no `stats_known`, declared state, failed run, incomplete run) **and the baseline is left untouched** |
+| `TestR431_OrdinaryRetentionIsSilent` | 69→60 silent · 10→7 silent · 10→5 silent (exactly half is not *more than* half) · 69→34 fires |
+| `TestR431_EscalationOnlyLatch` | a continuing deletion pages once; a clean sweep re-arms; a later deletion fires again |
+| **`TestR431_RealHistoryProducesZeroAlarms`** | **9 009 real points, 2 customers → ZERO alarms.** The acceptance step. |
+
+Full hub suite green (`go build ./... && go vet ./... && go test ./...`).
+
+**Red-proofs, by name:**
+
+1. **Threshold** — `snapshotDropFraction` 0.5 → 0.99: `TestR431_FiresOnAMassDeletion` **FAILS**
+   (`want exactly 1 …, got 0`).
+2. **`StatsKnown`** — guard removed: `TestR431_SilentWhenNotTrustworthy` **FAILS**
+   (`stats_known absent …: must NOT alarm; got 1`) **and the real-history replay FAILS too**.
+3. **Escalation-only** — latch removed: `TestR431_EscalationOnlyLatch` **FAILS**
+   (`a CONTINUING deletion must not re-page …; got 2 alarms`).
+
+## The live firing
+
+**Negative control first**, because an allowlist that accepts everything proves nothing:
+
+```
+POST /api/v1/event  event_type=zzz_not_allowlisted_r431  → HTTP 400  "Invalid event_type: zzz_not_allowlisted_r431"
+POST /api/v1/event  event_type=offsite_snapshots_dropped → HTTP 200  {"ok":true}
+```
+
+Hub log: `[INFO] Event from demo-hp: offsite_snapshots_dropped (error) — …` then
+`[INFO] Operator email sent for demo-hp/offsite_snapshots_dropped`.
+
+**Routing, from the live DB — this is the operator-only claim, with a control:**
+
+```
+788|customer|skipped|operator_only
+787|operator|sent|
+```
+
+Other types do reach customers (`escrow_blob_served|customer|41`, `whole_guest_backup_failed|customer|28`),
+so "operator-only" is a real distinction here rather than everything being operator.
+
+**The mail, rendered from the hub's own `FormatOperatorEmail`:**
+
+```
+SUBJECT: [Felhom] 🔴 demo-hp: offsite_snapshots_dropped
+Customer: demo-hp
+Event:    offsite_snapshots_dropped
+Severity: error
+Time:     2026-09-01 14:30 CEST
+Message:  Customer demo-hp: off-site backup count fell from 69 to 4 snapshot(s) in one report - more
+          than retention can explain. The daily Storage Box snapshots are read-only and still hold the
+          older copy, so this is recoverable file-by-file; it is NOT confirmed data loss. Check whether
+          a deletion ran on the box before restoring anything.
+Details:  {"previous_count":69,"current_count":4,"drop":65}
+Dashboard: https://hub.felhom.eu/customers/demo-hp
+```
+
+**⚠ THAT MAIL IS REAL AND VIKTOR RECEIVED IT. Nothing was deleted** — it is the required live firing,
+flagged as item 5 in `STATUS.md` so it is not acted on.
+
+## Not validated
+
+- **PROVEN-LIVE for the capability row.** The firing proves *delivery*, not that a genuine deletion is
+  caught on a live box. The capability row says **IMPLEMENTED**, with that gap written into it.
+- **Whether a NAMED snapshot can be entered** even though the directory does not list (ZFS permits
+  exactly that). One panel read from Viktor settles it — R-432.
+- **Whether a snapshot can be restored from.** Deliberately not attempted, in this session or the last.
+
+## No controller release, no golden
+
+**No controller change. No version bump there. No image. No golden owed.**
+`golden_currency_gate.py` exits 0; golden and fleet floor remain **0.232.0**.
+
+## Still open, named
+
+**R-430** (`unlock` reports success while the lock survives — **latent**: it bites only when the box
+loses delete, which is not happening here), **R-242's vouch half**, **R-402**, **R-409**, **R-401**,
+**R-412 leg 2**, **§11-D** (whether the two Hetzner services share an account), **R-427** (twelve open
+rows carrying a closed verdict), **R-432**.
+
+## Register
+
+**Before:** OPEN 181 · CLOSED 161. **After:** OPEN 181 · CLOSED 163.
+Corrected and closed **R-429**; shipped and closed **R-431**; filed **R-432**; re-scoped **R-95**
+(stays open, ranking to Viktor). Both closed rows were **moved into `CLOSED-ITEMS.md`** rather than
+left in the open register with a closed verdict — that pile is R-427 and I did not add to it.
+
+## Observations, and my own mistakes by name
+
+1. **A correct instrument aimed at the wrong subject produces a confident wrong answer, and controls
+   cannot save you from it.** My `.snapshots` probe had a good positive and negative control and was
+   still worthless. **FILED: R-429** — corrected there, with the cause named.
+2. **My mistake — I asserted a mechanism from a task brief without checking the vendor documentation**,
+   and reported "the safety net cannot be seen" to Viktor on that basis.
+   **NOT-A-FINDING: this is item 1 seen from the other side, already recorded in R-429 and as a
+   ruling in `CONTEXT.md`, so a second row would duplicate rather than add anything.**
+3. **My mistake — my first escalation-only test was HOLLOW and its red-proof passed.** It re-swept the
+   same report, so the baseline had already moved to the new count and the latch was never consulted.
+   Caught because red-proof 3 did **not** fail. Rewritten to drive a continuously falling count, which
+   is the only shape where the latch is load-bearing; the red-proof then failed correctly.
+   **NOT-A-FINDING: caught inside the session by the red-proof discipline doing exactly its job — a
+   test whose red-proof passes is not a test, and that is why they are run.**
+4. **My mistake — I queried the wrong table for the history.** The brief pointed at `host_reports`;
+   that is the *agent's* host report and carries no `offsite` object at all (8 434 rows, zero hits).
+   The controller's report lives in `reports`. **NOT-A-FINDING: found in one query by checking the
+   parse count instead of trusting the pointer; the measurement in the changelog and the fixture both
+   come from the right table.**
+5. **My mistake — I posted the live firing to `/event` instead of `/api/v1/event`** and got HTTP 302
+   for *both* the control and the test. **Two identical results are an instrument fault, not two
+   findings** — the same shape as yesterday's `sftp -p`. **NOT-A-FINDING: corrected in one command
+   once I read the router's `TrimPrefix`; no wrong conclusion was recorded.**
+6. **A sub-account cannot see inside the snapshot tree, so recovery is operator-only today.**
+   **FILED: R-432** — and it decides whether R-95's remedy can ever be product-driven.

REPORT-r87-spike.md

@@ -0,0 +1,229 @@
+# REPORT — R-87 spike: can the off-site copy be restore-tested without a person? (2026-08-31)
+
+Written as `REPORT-r87-spike.md`, not `REPORT.md`: this repo's `CLAUDE.md` says the shared report is
+overwritten and two sessions clobber each other.
+
+**Findings document (the deliverable):** `documentation/audits/SPIKE-restic-restore-test-2026-08-31.md`
+**Evidence:** `documentation/audits/evidence-spike-restic-restore-2026-08-31/` — 31 files.
+
+---
+
+## 1. Baselines, re-checked at the start
+
+| repo | `main` @ | version | matched the task's stated baseline? |
+|---|---|---|---|
+| `felhom-controller` | `2d802d75e88616d86cbade8a0e16965c2b85771c` | v0.230.0 | yes |
+| `felhom.eu` | `dddcc808be95d1c89b276b4d791491bad3c96bba` | — | yes; clean tree, `HEAD == origin/main` |
+| `felhom-agent` | `058b945` | v0.130.0 | yes |
+
+## 2. Part 1 — the register correction
+
+- **The mis-filing commit:** `ef6ac6f`, 2026-08-22, *"One register, enforced by a gate; closed work
+  compressed into siblings (R-376..R-378)"*. Established by `git log -S '**R-87**'` on both register
+  files — it is the only commit that added the row to `CLOSED-ITEMS.md` and the only one that removed
+  it from `OPEN-ITEMS.md`. **The history settled it; no guess was needed.**
+- **It is a survivor of R-378, not a separate incident.** R-378 records six rows moved wrongly by that
+  same sweep and restored in the same session. R-87 is a **seventh it missed**, and it escaped because
+  its state cell led with `READY` and carried the word `closed` later, describing a different row.
+- **My own count, reproduced:** **1** mis-filed row under the leading-verdict predicate. The same scan
+  convicts **3** if it reads the whole state cell (R-224 and R-260 are false positives — long prose
+  verdicts containing "open"/"OPEN") and **144** if it reads the whole row. The task author's count of
+  one is confirmed, and only under the predicate R-378 argues for.
+- **The gate:** `scripts/closed_register_gate.py`. Two rules. **Red-proof rule 1:** a planted `READY`
+  row convicts by name, rc=1; removing it leaves the file byte-identical. **Red-proof rule 2:** a
+  planted duplicate id convicts, rc=1. **Negative control:** run against the files *as pushed*
+  (`HEAD:`), it convicts R-87 at L72 and R-398 at L139, rc=1. **Registered LAST**, as the 12th gate in
+  `repo_gates.py`, after it was green.
+- **Also corrected:** `R-398` had a row in both registers (a deliberate cross-reference stub). Now
+  prose beneath the table.
+
+## 3. Q1–Q7, one paragraph each
+
+**Q1 — restic 0.14.0**, `go1.19.8`, Debian bookworm 12.15, from the running container. The four source
+comments asserting 0.14.0 are **confirmed**. Method: `docker exec felhom-controller restic version`.
+
+**Q2 — `--verify` exists and is NOT a content check.** `restic restore --help` lists
+`--verify  verify restored files content`; positive control `--target` = 1 hit, negative controls
+`--delete/--dry-run/--overwrite/--sparse` and a nonsense string = 0 hits each. Neither `--verify` nor
+`--no-lock` appears anywhere in the controller source (`grep -rn` rc=1, with `--json`/`--target` as
+the positive control). **Red-proof:** one byte changed in a restored 160 MB tar with size and mtime
+preserved — `restore --verify` **passed clean, rc=0**. Verify took **131 ms** on a 213 MB / 7-file
+tree, which cannot be hashing. A size or mtime mismatch causes a silent **re-download**, not a
+failure. **So restic cannot tell us a restore produced correct files.**
+
+**Q3 — no reference for "correct" exists today.** `restic ls --json` file nodes in 0.14.0 carry no
+content hash. The recovery unit's `manifest.json` hashes three config files — **4 918 B of a
+213 231 242 B unit, 0.0023 %** — and not the DB dump or the volume tars. A planted sentinel is a drill
+technique and does not transfer; the live data drifts. **What the manifest CAN answer is
+completeness**, through the existing `unitCarriesData` (`r403_hollow.go:40`), with no new metadata.
+Filed as R-409.
+
+**Q4 — ~4 s per app, 25 s for all eight, cheaper than the weekly check.** Through the product's own
+path: docmost unit 9 s, kimai full 11 s. Raw restic, all 8 snapshots / **774 378 123 B logical** back
+to back: **25 s**, individual times 2 253–3 978 ms *regardless of size* (185 KB → 2.25 s, 213 MB →
+3.20 s). The cost is per-snapshot round-trip plus ≈ 1 s per 200 MB. Peak scratch = the app's full
+logical size, 213 272 202 B for the largest. The restic cache is **1.1 MB** (index only) and does not
+hide the cost: `--no-cache` 5 423 ms vs cached 3 198 ms, trees byte-identical. **Against R-359's
+35.0 s / 39.2 s: the same 100 % check re-measured today is 40 257 ms — so restore-testing the whole
+box costs LESS than one weekly check.** Extrapolation to 10×/100× is in the findings doc, **labelled
+as extrapolation**, with scratch space named as the constraint that binds before time does; the
+single-store hole is R-401's.
+
+**Q5 — skip-if-busy stays right, and a bigger thing is wrong.** Scheduler registrations read off the
+box (CEST): db-dump 02:30, tier2 03:30, **offbox-backup 04:15 (2m52s measured)**, abandon-sweep 05:10,
+**offsite-integrity 06:00 (40.3 s)**. A 25 s hold is seconds, not minutes, and there is an empty gap
+04:18–06:00. **But `RestoreOffboxScratch` takes no `acquireRunning` at all** — nine non-test callers,
+it is not one — while `offbox_integrity.go:28` asserts *"Every off-site operation takes
+`acquireRunning`"*. `restore_wizard.go:174` records the same fact independently. Filed as **R-408**.
+
+**Q6 — the restore itself writes nothing; the product writes anyway; and `check` writes a lock.**
+Observed with a lock sampler and an argv sampler, both inside the container, the repo URL redacted at
+source. **Positive control:** across the product's integrity run the repo went `locks=0` →
+`locks=1 id=81fd4d42…` for nine consecutive samples → `locks=0`. **The same instrument saw zero locks
+across two restores**, so `restic restore` in 0.14.0 does not lock. Observed argv for one restore:
+`snapshots latest --tag <app> --json`, then **`unlock`**, then `restore <id> --target …`. The middle
+one is `unlockStale` (`offbox_restore.go:289`), unconditional, a **delete verb**. **§5's lead was
+right in direction and wrong in mechanism** — the write is `unlockStale`, not `resticStep`'s
+escalation. **The constraint IS satisfiable:** `--no-lock` + skipping `unlockStale` writes nothing,
+and both mechanisms exist in 0.14.0 unused. `offbox_integrity.go:255`'s *"It NEVER writes to the
+repository"* is filed as **R-407**. **Neither was fixed** — §7 forbids it.
+
+**Q7 — one of five.** R-353 (a *local* restore path) **no**; R-354 (no volume-replay leg, after the
+scratch) **no**; **R-356 (refused every driveless app) YES — five of eight apps on this box would
+have fired it on the first night**; R-358 (needs a part-way failure) **no**; R-403 (destroyed a
+*local* copy) **no as filed, yes for the shape**. **The honest verdict is "few", and it points
+elsewhere:** `check` proves the stored bytes are the stored bytes, never that we stored the *right*
+thing. A hollow unit backs up, checks at 100 % and restores cleanly, and recovers nothing — R-403,
+measured in bytes nine days ago. Nothing asks that question on any tier.
+
+## 4. Recommendation
+
+Three options with costs and a do-nothing outcome are in the findings document. **I would pick option
+C — the narrow test:** one app a night, restored to scratch, checked against its own `manifest.json`,
+scratch deleted, the snapshot recorded as the proof. ~4 s and ≤ 213 MB per night; catches R-356 and
+the R-403 class; needs no new metadata. **It must use `--no-lock`, skip `unlockStale`, and take
+`acquireRunning`** — all three established by this spike. **Options A (do not build) and B (scheduled
+attended drill) were considered explicitly and are argued in the document; B is the weakest, because
+it is what already happens.** **R-87 should be RE-SCOPED, not built as written — and that is Viktor's
+call**, so the row stays open carrying the verdict, and `STATUS.md` item 4 asks it in plain words.
+
+## 5. Evidence
+
+`documentation/audits/evidence-spike-restic-restore-2026-08-31/`, 31 files, numbered by question.
+**Every file was pulled off the box before any teardown** (R-320) — including the two in-container
+sampler logs, which were `cat`ed to DooPlex before the container `/tmp` was cleared.
+
+## 6. Probes removed — all three layers, and none of them is "nothing was created"
+
+| layer | created | after teardown |
+|---|---|---|
+| PVE host `/root` | 6 scripts + one 0600 password file | `ls \| grep` → nothing |
+| guest 9201 `/root`, `/tmp` | 9 files | grep → nothing |
+| container `/tmp` | 5 files + 2 run-flags | `/tmp` lists empty; no restic process left |
+
+**Scratch directories:** the four created by this session's restores (`docmost`, `kimai`,
+`privatebin`, `opengist`) were removed. Three (`bookstack`, `calibre-web`, `paperless-ngx`) pre-date
+this session and were **left alone**. The local password copy was `shred -u`'d.
+
+**Two state changes recorded rather than hidden:** the integrity check run as the lock positive
+control **recorded its verdict** (`last_integrity_check` → `2026-08-31T13:41:28Z`, depth `structure` →
+**`100%`**, due-ness advanced 7 days), and four restores plus two logins appear in the controller log.
+**Nothing was written to the off-site repository by hand.**
+
+## 7. Register
+
+| id | action |
+|---|---|
+| **R-87** | **moved back to `OPEN-ITEMS.md`** (verbatim from `ef6ac6f^`, beside R-95), then updated with the spike verdict and a re-scope proposal |
+| **R-398** | de-tabled in `CLOSED-ITEMS.md`; the open row is the record |
+| **R-404** | bypass count corrected six → **seven** (this session's Part 1 push) |
+| **R-405** | filed + **CLOSED** — the mis-file, the reproduced count, the gate |
+| **R-406** | filed — two findings share the id R-133 |
+| **R-407** | filed — `restic check` takes a lock; the comment says it never writes |
+| **R-408** | filed — `RestoreOffboxScratch` takes no `acquireRunning` |
+| **R-409** | filed — the unit manifest hashes 0.002 % of the unit |
+
+**Register size:** `OPEN-ITEMS.md` **165 → 171** rows (+R-87 restored, +R-405..R-409); `CLOSED-ITEMS.md` **153 → 151** (−R-87, −R-398).
+Ceiling **R-404 → R-409**.
+
+`python3 scripts/unproven.py --summary`: 55 claims, walked 20 / partial 17 / built 14 / missing 4,
+**NOT WALKED 35 of 55 — unchanged by this session**, which shipped no product claim.
+
+## 7b. Golden 0.230.0 — baked, vouched, delivered (second half of the session, on request)
+
+**This was NOT part of the spike and is reported separately so the two are not confused.** Asked for
+after the spike closed; the spike itself still changed no product code.
+
+| | |
+|---|---|
+| `GOLDEN_SHA256` | `9287f7cef5f13166276e8406005e3f28004004510c5184f1c1c7377f7aafad2e`, 657 873 700 B |
+| markers | `docker OK (overlay2` 1 · `including mount point` rootfs 1 + mp0 1 · `upload OK (HTTP 201)` 1 · `excluding` 0 · `FATAL` 0 — counted on the **committed** log |
+| three readers agreed | the bake's own print, the **round trip** of the published bytes, and the hub's Day-0 dropdown reading Gitea on a different code path |
+| the artifact names its own controller | `tar --zstd -xOf golden.tar.zst ./etc/felhom-controller-image` → `felhom-controller:0.230.0`, with 19 382 entries under `var/lib/felhom/docker/` |
+| vouch | `golden_version` 0.229.0 → **0.230.0**; `agent_version` 0.130.0 and `min_agent` 0.129.0 **unchanged** — v0.230.0's `MinAgent` is 0.129.0, and 0.129.0 ≤ 0.130.0 so this is not the R-216 shape. Re-read from the page; `golden_behind_fleet` confirmed absent |
+| floor | `min_controller_version` 0.229.0 → **0.230.0**, a separate setting, done on the operator's explicit answer |
+| **the unattended proof** | `demo-felhom` was on **0.229.0 — the R-403 build** — and moved itself: `controller-swap: image file written` 16:21:30 → `controller-swap: new controller healthy` 16:21:40 CEST. Both boxes now 0.230.0, healthy |
+| pre-gates | 404 pre-gate passed; token-leak grep **0** on the committed log **and 1** on a seeded throwaway copy, so the zero is earned. Unit properties grepped for the token: **0**, positive control **1** |
+| teardown | `pct destroy 9100 --purge`, `shred -u` after the log was copied out, `poweroff`, qemu confirmed exited with `ps -eo comm` (not `pgrep -f`, which self-matches), disk back to `virgin` |
+
+Full record: `documentation/tests/golden-0.230.0-2026-08-31/README.md`.
+
+**One honest gap vs. the 0.229.0 precedent:** the bake script's sha256 was **not** compared across
+the hop, only recorded on DooPlex (`7b0fb5cf…73b6a1`). A corrupted `scp` would have failed the bake
+rather than produced a wrong golden — but that is an argument, not a measurement.
+
+**And the gate that flagged all this has a hole, found while it went green:** `golden_currency_gate.py`
+matches a **directory name** (`scripts/golden_currency_gate.py:89,123`). I created
+`documentation/tests/golden-0.230.0-2026-08-31/` before the bake finished, and the gate would have
+passed at that moment. Filed as **R-410**.
+
+## 8. Controller code, and the golden debt as it now stands
+
+**The spike changed no controller code**, and that remains true — the golden bake ships the image
+that was already released as v0.230.0, unchanged.
+
+## 8. No controller code changed and no golden is owed
+
+`felhom-controller` and `felhom-agent` were **read only** for the spike. No version bump, no build,
+no deploy. **The golden debt — v0.230.0 released with the newest bake at 0.229.0 — was already red at
+`dddcc80` before this session started** and belonged to that release, not to the spike;
+`golden_currency_gate.py` was the only failing gate throughout the spike. **It was then PAID on
+request** (§7b): the gate now exits 0, and the two register-only pushes below were the last that
+needed a bypass.
+
+**`git push --no-verify` was used, twice, for exactly that reason** — records-only pushes meeting the
+golden gate. That is R-404's subject and the count is updated in its row.
+
+**CI is RED for both of this session's pushes, and I checked rather than assumed.** Pulled by run id
+from `gitea.dooplex.hu/api/v1/repos/admin/felhom.eu/actions/tasks`:
+
+| run id | run_number | head_sha | status |
+|---|---|---|---|
+| 451 | 280 | `66156c619f` | success |
+| **456** | **281** | **`dddcc808be`** | **failure — the commit this session STARTED from** |
+| **458** | **282** | **`6e550aedd3`** | **failure — this session's Part 1** |
+| **459** | **283** | **`130f7a6eba`** | **failure — this session's spike commit** |
+| **460** | **284** | **`32a4c35c9c`** | **failure — the CI-verdict amendment** |
+| **461** | **285** | **`2263245cf2`** | **SUCCESS — the golden bake commit** |
+
+CI runs the same `repo_gates.py` entry point, so it fails on `golden_currency_gate.py` exactly as the
+pre-push hook did. **Run 281 is the proof that it is not mine:** it is the previous session's commit,
+pushed before this session began, and it is already red. Nothing else in the suite fails at any of the
+three commits. **It resolved exactly there: run 285, the golden-bake commit, is GREEN** — the first green run since
+`66156c619f`, and the first push this session that the pre-push hook let through unbypassed
+(`pre-push [felhom.eu]: gates OK - push proceeding`). All 13 gates pass.
+
+## 9. Observations — noticed, not acted on
+
+- `paperless-ngx` has an off-site snapshot under that tag and none under `paperless`; `filebrowser`
+  has none at all (it is infrastructure, so that may be correct). Not chased.
+- `CLOSED-ITEMS.md` rows **R-399** and **R-400** supply two columns where the table declares four —
+  they render with no `Shipped` and no `Evidence`. The new gate warns rather than convicts, because an
+  empty state cell is not an open state word.
+- Two rows (**R-309**, **R-351**) carry a `|` inside their body, shifting their own cells. Named as
+  the gate's first residual hole.
+- The controller image has **no `ps` and no `python3`**. `/proc/*/cmdline` is the substitute that
+  works, and it is worth knowing before writing any probe that runs in there.
+- The guest scheduler logs in **CEST**, not UTC — `offbox-backup scheduled for 2026-09-01 04:15 CEST`
+  against a `last_run` of `02:17:57Z`. Consistent, and the opposite of what the project memory says
+  about guest time.

REPORT-r95-spike.md

@@ -0,0 +1,127 @@
+# REPORT — SPIKE R-95: can the box be stopped from deleting its own off-site history? (2026-09-01)
+
+## Q1 first, and it raises the urgency rather than lowering it
+
+**The safety net cannot be seen from the box, on either machine, so the seven-day bound is
+unverified — and unverifiable from the product side.**
+
+Measured over each box's own SFTP credential, read-only, with controls that passed first:
+
+| probe | demo-hp (sub-account A) | demo-felhom |
+|---|---|---|
+| positive control — account home | lists `.ssh`, `<repo>` | lists `.ssh`, `<repo>`, `<repo>.orphaned-20260810` |
+| negative control — bogus name | `not found` | `not found` |
+| `./.snapshots` | **`not found`** | **`not found`** |
+| `<repo>/.snapshots` | `not found` | `not found` |
+| `/` | `Permission denied` (jailed) | same |
+
+**Either no snapshots exist, or a sub-account cannot see them. The box cannot tell which, and neither
+is a reprieve** — a snapshot the box cannot see is one it cannot restore from, so recovery is an
+operator act at the Hetzner panel, not a product capability.
+
+**The register's claim rests on nothing that was ever checked.** `OPEN-ITEMS.md:233` is a row with
+**no R-number**, its "confirm tomorrow" was **2026-07-27** (36 days), and the `DUE-CHECKS` block built
+for exactly this (R-341) is **empty**. R-95's own text says "Mitigation now ARMED"; that word is
+**withdrawn** pending R-429.
+
+**The task expected Q1 might bound the exposure to seven days. It does not.** I stopped at the §11-D
+fence rather than answering it with the provider token.
+
+## Q1–Q7, one answer each
+
+| Q | answer |
+|---|---|
+| **Q1** | **NO / unknowable from the box.** Measured on both machines with controls. → R-429 |
+| **Q2** | **Ten verbs, not nine, and TWO `forget --prune` sites.** `check` (`offbox_integrity.go:316`) is missing from the task's list; `dump` is not a verb (`offbox_progress.go:185` is a phase constant) — withdrawn. Delete-capable: `forget`/`prune` (`offbox.go:1388` **and `:1759`**), `unlock` (`:746`, `:768`). |
+| **Q3** | **No. DOCUMENTED** from `hub/internal/hetznerapi/hetznerapi.go:38-45`: `AccessSettings` has five booleans and **`readonly` is the only permission axis**. No append-only. Corroborated by existing state, no new action: demo-felhom's home holds a `<repo>.orphaned-20260810` directory produced by a controller **rename**, which is delete-class. |
+| **Q4** | **The PBS shape does not transfer.** PBS is a server that can refuse; a Storage Box is a filesystem that **runs nothing**, so there is no far end to move retention to. Four candidates costed; each concentrates a delete-capable credential. **R-191's trap is doubled here** — both `forget` sites must be disarmed in the same change or every successful backup reports failure. |
+| **Q5** | **NOT the blocker — measured.** Under a faithful append-only model (both controls passed), a stale lock does **not** wedge the store: `backup`, `check`, `snapshots --no-lock` and `restore --no-lock` all succeeded. **But `unlock --remove-all` printed `successfully removed locks` while the lock survived** → R-430. The crash-lock (foreign hostname) window is **UNKNOWN**. |
+| **Q6** | **Reachable. MEASURED:** `rest:` gives a connection error where the control `banana:` gives `invalid backend` — **restic 0.14.0 speaks REST**. Append-only is a **rest-server** flag; `restic help` contains zero occurrences of "append". Needs a machine in the recovery path (**ep0 is protected — architecture change**) and either a mount in the hot path or moving every customer's history. |
+| **Q7** | **Nearly free. MEASURED:** `snapshot_count` already reaches the hub (`report/types.go:131` → `backup_card.go:118`) and **the hub APPENDS reports** (`store.go:968` INSERT; read is `ORDER BY id DESC LIMIT 1`), so the history to compare against is already on disk. No box change, no credential, no new service. |
+
+## The four options, ranked
+
+**My pick: answer Q1 today (Viktor, ten minutes) → build 4 → then 2. Defer 3.**
+
+1. **Do nothing — not acceptable as it stands.** It used to mean "bounded to seven days". Q1 shows
+   that sentence is unsupported. £0, no evenings, and an unbounded exposure on the tier holding the
+   customer's documents.
+2. **Copy the PBS shape — worth doing, smaller than it sounds.** Q5 removed the fear that it would
+   wedge the store. But Q3 means the credential still *can* delete; the box would merely stop using
+   it. That is discipline, not a guarantee, and a compromised guest is unaffected. One evening + a new
+   home for retention.
+3. **Change the transport — the only real prevention, and not yet.** Q6 says it is reachable. It costs
+   an always-on service in the recovery path and a protected machine's architecture. Choosing it
+   before Q1 is answered is the wrong order.
+4. **Detect instead of prevent — cheapest by a wide margin, do this first.** Q7 measured that the
+   material already exists. Converts "we would never know" into "we know tomorrow". Third instance
+   this week of *proving beats preventing when preventing is expensive*.
+
+## What I could not measure, and what would settle it
+
+| unknown | what would settle it |
+|---|---|
+| **Do snapshots exist on the Storage Box?** | the Hetzner panel or `size_snapshots` via the API — **fenced by §11-D; stopped and left for Viktor (R-429)** |
+| Whether the live API exposes any permission the Go struct omits | the provider token — **same fence** |
+| The **crash-lock** case (a lock whose hostname restic cannot match, non-stale for ~30 min) | a lock captured from a container with a **different hostname**, replayed against an append-only endpoint. My model could not build it — the captured lock carried this container's own hostname |
+| Whether `unlock` reports success because it removed zero locks by design, or because it never checked | read restic 0.14.0's unlock source, or re-run with `--verbose` |
+| Whether a Storage Box snapshot can be **restored** | deliberately not attempted — named as the next question, per the brief |
+
+## Register
+
+**Before:** OPEN 179 · CLOSED 161. **After:** OPEN 181 · CLOSED 161.
+**Filed R-429** (the unconfirmed snapshot mitigation, and the id-less row), **R-430** (`unlock` lies
+about success). **R-95 updated** with the verdict and kept OPEN; its word "ARMED" withdrawn.
+**Docs:** `07` §8 row 10 and §10.2 gained the verdict (**row 10's status deliberately NOT moved**);
+§11-D records that the fence was reached again and held; `STATUS.md` carries one plain-language item.
+
+## Compliance
+
+- **No code changed. No version bumped. No image built. No golden owed.** `golden_currency_gate.py`
+  exits 0; golden and floor remain **0.232.0**.
+- **No delete verb was issued against any live store** — no `forget`, `prune`, `unlock` or `init`.
+  Every live-store interaction was an SFTP `ls`.
+- **`ep0`, DooPlex and Peti's box were not touched at all**, not even read — the two demo boxes were
+  the only machines used.
+- **The Hetzner API and control panel were not called.**
+- **Scratch resources:** a throwaway local restic repo under `/tmp/r95s` (and `/tmp/r95scratch` in the
+  first attempt) inside the controller container on demo-hp, 60 MB, plus three probe scripts. **All
+  removed**, verified by the scripts' own teardown output (`scratch: gone`). Nothing was created on
+  any Storage Box.
+
+## Observations, and my own mistakes by name
+
+1. **The register calls a mitigation ARMED that has never been confirmed, in a row that cannot be
+   cited because it has no id, with the dated-check mechanism sitting empty beside it.**
+   **FILED: R-429.**
+2. **`restic unlock --remove-all` reports success on a deletion that did not happen**, and the
+   crash-lock self-heal is built on it. **FILED: R-430.**
+3. **The task's own verb list was missing `check` and included `dump`, which is not a verb**, and it
+   names one `forget --prune` site where there are two. **NOT-A-FINDING: the brief invited me to
+   confirm the list myself, which is what this is; both corrections are in the spike document and the
+   second one is carried into R-95's row, because disarming one site and not the other reproduces
+   R-191 exactly.**
+4. **My mistake — my first Q5 model proved nothing.** I used `chmod a-w` and ran restic as **root**,
+   which ignores permission bits, so every verb succeeded and I nearly recorded "no wedge" on a test
+   that tested nothing. Rebuilt as a sticky directory with a root-owned lock and restic run as
+   `nobody`, with two controls. **NOT-A-FINDING: caught inside the same session by the result being
+   too clean; the corrected model is the one reported, and the first is described so nobody repeats
+   it.**
+5. **My mistake — my first `sftp` probe used `-p` for the port**, which `sftp` reads as "preserve", so
+   the port became the destination and all three probes returned identical usage errors. **The
+   controls are what exposed it** — a positive and a negative control failing the same way is an
+   instrument fault, not a result. **NOT-A-FINDING: a flag error of mine, corrected in one command;
+   it is recorded because the failure mode it demonstrates — three identical errors reading as three
+   findings — is the one this project keeps paying for.**
+6. **My mistake — I wrote the §8 verdict onto row 4 instead of row 10.** The anchor text I matched
+   appears in both rows and I replaced the first occurrence. Caught by checking the line number,
+   reverted from row 4 and applied to row 10, both verified by grep. **NOT-A-FINDING: an editing error
+   of mine, corrected within the session and verified in both directions, so no wrong claim ever
+   reached a push.**
+7. **My mistake — I put two escaped pipes inside a register row**, which makes it a five-column row in
+   a three-column table and would have made it unreadable to `closed_register_gate.py` — **the exact
+   defect I fixed in that gate yesterday.** Caught by counting pipes before committing. **NOT-A-FINDING:
+   corrected before the push; recorded because I introduced the same shape twice in two days.**
+8. **The task's baseline table lists `felhom-agent` at `058b945`; it is at `4586f0f`.**
+   **NOT-A-FINDING: that is my own push from the previous session, so the table was stale rather than
+   wrong about anything that matters here; the agent repo was not touched by this spike at all.**

REPORT-register-and-floor.md

@@ -0,0 +1,333 @@
+# REPORT — dated checks that bite, the floor raise on the record, the snapshot that covers less (2026-08-18)
+
+**Three pieces of bookkeeping, no machine put at risk. The hub was READ ONLY throughout.**
+
+**The headline is that Part 3's premise was wrong.** The floor raise was *not* a no-op: it moved a
+live customer box nine seconds after the save. That is the whole reason the task said to read it back
+rather than assume it. **R-343 is therefore filed OPEN, not CLOSED**, per the task's own condition.
+
+---
+
+## 1. Confirmed baselines
+
+| item | value |
+|---|---|
+| felhom.eu `main` @ start | `f267bc047f198de4cb600068fdd8bcef557cff20` — matches the sheet |
+| clean tree at start | yes; `HEAD == origin/main` in felhom.eu, felhom-controller, felhom-agent |
+| `scripts/` version IN | `felhom-host-install.sh v1.28.0` (CHANGELOG head) |
+| `scripts/` version OUT | `due_checks_gate.py v1.0.0` (new head entry) |
+
+## 2. Files created / modified
+
+**Created:** `scripts/due_checks_gate.py`, `scripts/test_due_checks_gate.py`,
+`REPORT-register-and-floor.md`.
+**Modified:** `scripts/repo_gates.py` (registration + docstring), `scripts/CHANGELOG.md`, `CLAUDE.md`,
+`CONTEXT.md`, `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md` (block + R-342 + R-343),
+`documentation/runbooks/publish-train-rules.md`.
+Commit hashes are in §9.
+
+## 3. Tests — 37 assertions, and a red-proof that caught my own test
+
+`python3 scripts/test_due_checks_gate.py` → **passed: 37, failed: 0**, groups A–G.
+
+### Red-proof 1 — the boundary. **It failed usefully: it exposed a HOLLOW assertion of mine.**
+
+**Mutation:** `due_now = [... if r[1] <= today]` → `< today`.
+**First run, before the fix:** Group C reported
+
+```
+  PASS  C: due TODAY exits 1  rc=1          <-- passed, and should NOT have
+  FAIL  C: says DUE TODAY rather than overdue
+```
+
+**The `rc == 1` assertion passed for the wrong reason.** With `<`, a row dated exactly today falls
+into neither `due_now` (`<`) nor `pending` (`>`), so `min(pending, …)` raised
+`ValueError: min() iterable argument is empty` and the **traceback** exited 1. Confirmed directly:
+
+```
+  File ".../due_checks_gate.py", line 232, in main
+    nearest = min(pending, key=lambda r: r[1])
+ValueError: min() iterable argument is empty
+RC=1
+```
+
+**An exit code alone cannot distinguish a verdict from a crash.** Two fixes, both kept:
+
+1. the test now asserts `DUE-CHECKS GATE FAILED` is in the output **and** `Traceback` is not, plus a
+   new `test_c_gate_never_ends_in_a_traceback` across overdue/future/empty inputs;
+2. the gate returns **2 (INCONCLUSIVE)** with a message if the partition is ever broken again,
+   because a crash is never a verdict.
+
+**Re-run after the fix — the mutation now bites properly:**
+
+```
+  FAIL  C: due TODAY exits 1 (boundary is <=)  rc=2
+  FAIL  C: exits 1 as a VERDICT, not a traceback
+  PASS  C: did not crash
+  FAIL  C: says DUE TODAY rather than overdue
+passed: 34   failed: 3
+```
+
+**Mutation reverted**, verified by `grep -n "MUTATED"` returning nothing and the `<=` line restored.
+
+### Red-proof 2 — the missing-block path
+
+**Mutation:** the missing-block branch `sys.exit(2)` → `sys.exit(0)`.
+**Seen failing:**
+
+```
+  FAIL  E: missing block exits 2 (NOT 0)  rc=0
+passed: 36   failed: 1
+```
+
+**Reverted**, `sys.exit(2)` restored on that branch.
+
+## 4. The gate's real output in all three states
+
+**Overdue (fixture, today=2026-08-20):**
+
+```
+DUE-CHECKS GATE FAILED: 1 dated check(s) are due or overdue as of 2026-08-20 (UTC).
+
+  R-341    due 2026-08-19   1 day(s) OVERDUE
+           measure: ep0 proxy fd count + ESTAB/CLOSE-WAIT split; PID must still be 551655
+           the command and its preconditions are in the R-341 row of documentation/backlog/OPEN-ITEMS.md
+
+Take the measurement, record the result in that R-row, then remove the row from the
+DUE-CHECKS block. Moving the date instead is allowed — state the reason in the R-row.
+NOTE: this gate fires on a PUSH, not on the date; it may be later than the date.
+```
+
+**Pending / the LIVE run against the real register today (these are the same run):**
+
+```
+due-checks gate OK — 2 dated check(s) pending, none due yet.
+  today (UTC): 2026-08-18
+  nearest: R-341 due 2026-08-19 (in 1 day(s)) — ep0 proxy fd count + ESTAB/CLOSE-WAIT split; PID must still be 551655
+  (fires on the next PUSH after a date passes, not on the date itself — by design)
+```
+
+## 5. The runner's output
+
+```
+  site               OK            (exit 0)
+  hostinstall        OK            (exit 0)
+  hub-confirm        OK            (exit 0)
+  manifest-bearer    OK            (exit 0)
+  reuse-refs         OK            (exit 0)
+  instructions       OK            (exit 0)
+  golden-currency    OK            (exit 0)
+  wire-contract      OK            (exit 0)
+  hub-copy           OK            (exit 0)
+  due-checks         OK            (exit 0)
+all felhom.eu gates OK
+```
+
+Group G asserts registration by **running the runner** and matching `due-checks` in its output, never
+by grepping `repo_gates.py`'s source — a commented-out entry still contains the string.
+
+## 6. Part 3's five reads — evidence, not summary
+
+### READ 1 — the live floor, from the store
+
+```
+  artifact_agent_version     = '0.129.0'    (updated 2026-08-18 11:00:59)
+  artifact_golden_version    = '0.216.0'    (updated 2026-08-18 11:00:59)
+  artifact_min_agent         = '0.129.0'    (updated 2026-08-18 11:01:00)
+  min_controller_version     = '0.216.0'    (updated 2026-08-18 12:36:58)
+```
+
+**The raise landed**, so Part 3 proceeded. Read from `hub_settings`, not the form.
+
+### READ 2 — per-customer overrides
+
+```
+  demo-felhom      status=active     override=''         config_version=12
+  demo-hp          status=active     override=''         config_version=5
+  drill-r50        status=blocked    override=''         config_version=1
+  peti-felhom      status=active     override=''         config_version=6
+  tester-1         status=active     override=''         config_version=1
+  -> 0 customer(s) carry a non-empty override
+```
+
+**Zero overrides**, so the global applies to everyone and no box hides behind a lower one.
+
+### READ 3 — every box's controller version. **Two are below the floor.**
+
+```
+  demo-felhom      controller='0.216.0'    last_report=2026-08-18 13:07:07
+  demo-hp          controller='0.216.0'    last_report=2026-08-18 13:01:34
+  drill-r50        controller='0.213.0'    last_report=2026-08-12 15:33:25
+  peti-felhom      controller='0.115.0'    last_report=2026-07-15 08:39:00
+```
+
+**The two REPORTING boxes are both at 0.216.0, at the floor.** The other two are below it and neither
+is a reporting box: `drill-r50` is `status=blocked`, last heard from six days ago, powered off and
+reverted; `peti-felhom`'s host row was deleted on 2026-07-15. Reported here rather than as a
+footnote, per the task's edge-case rule.
+
+*(Note: `guests.controller_version` is empty for every guest — the hub carries the controller version
+on `reports.controller_version`, not on the guest row. The first query I wrote read the guest field
+and would have reported "unknown" for every box.)*
+
+### READ 4 — directives and holds
+
+```
+2026/08/18 14:36:58 [INFO] Global controller-version floor set to "0.216.0"
+```
+
+**No `managed floor HELD` line exists** — searched over 24 h of pod logs. (Hub log lines are CEST;
+the DB stores UTC, hence 14:36:58 here and 12:36:58 above — the same instant.)
+
+### READ 5 — **THE FINDING: a controller DID auto-update after the raise**
+
+```
+2026-08-18 12:37:07 | demo-felhom | controller_updated | Controller frissítve: 0.214.0 → 0.216.0
+2026-08-18 12:37:12 | demo-felhom | controller_started | Controller elindult (0.216.0)
+```
+
+and the version trail confirms it:
+
+```
+   2026-08-18 11:14:55  controller=0.214.0
+   2026-08-18 12:37:12  controller=0.216.0
+```
+
+**`demo-felhom` had been on 0.214.0 since 2026-08-12 16:44 and the floor raise pulled it to 0.216.0
+nine seconds after the save** — exactly the "acts immediately on the next report cycle" that
+`publish-train-rules.md` rule 2 documents and that the 2026-07-11 incident was filed for.
+`demo-hp` was already on 0.216.0 (hand-deployed 2026-08-14 08:31) and did not move.
+
+**No error, warning or critical event followed** — the update completed and the controller restarted.
+So: harmless in outcome, but **not a no-op**. "Every reporting box is at or above the floor" is true
+**because of** the raise, not independently of it.
+
+**R-343 is filed OPEN.** The task's closing condition was *all five reads clean and no directive
+served*; read 5 shows a live box moved. It went well, and a record that called it inert would mislead
+the next reader.
+
+## 7. `peti-felhom` — not contacted
+
+**The machine was not contacted in any way.** Sourced from the PETI register row, quoted:
+
+> *"a report from a deleted host 401s and is not persisted"*
+
+with its host row deleted `2026-07-15 08:56:22` (`host_deletions` id=1). It therefore cannot receive a
+floor directive and the raise cannot reach it. Its `reports` row still shows controller 0.115.0 from
+its last report on 2026-07-15 08:39:00 — a stale record, not a live box.
+
+## 8. The two `build-felhom-iso.sh` facts, confirmed in the script
+
+**(a) It is a BUILD-TIME gate.** `assert_golden_ge_floor()` is defined at **`:77`** and called at
+**`:267`**, in the build flow.
+
+**(b) It FAILS OPEN with a warning when its inputs are absent** — `:78-82`:
+
+```bash
+    local golden="${FELHOM_ASSERT_GOLDEN:-}" floor="${FELHOM_ASSERT_FLOOR:-}"
+    if [[ -z "$golden" || -z "$floor" ]]; then
+        log_warn "R-71 golden>=floor gate UNENFORCED — pass FELHOM_ASSERT_GOLDEN + FELHOM_ASSERT_FLOOR to enforce (golden='${golden:-unset}' floor='${floor:-unset}')"
+        return 0
+    fi
+```
+
+Both read as the task described. **No ISO rebuild is required:** the golden is fetched at first boot
+from the hub's manifest (0.216.0 — at the floor, not below it), and this gate governs *future* builds.
+
+
+## 9. Commits and CI
+
+| commit | contents |
+|---|---|
+| `0a5e9b14dc84ecfb179b2654d079c2f6d3f15fe2` | the gate, its tests, registration, both register rows, the block, and all §5 documentation |
+
+**CI run `355`, `head_sha 0a5e9b14d`, conclusion `success`** (started 2026-08-18T13:17:04Z). The
+previous run `354` on `f267bc047` was also green, so this run's green is attributable to this change
+rather than inherited from a red baseline — and per §13 of the task, a red run here would have been
+mine to own.
+
+**The push needed no `--no-verify`.** The pre-push hook ran `repo_gates.py --fast`, including the new
+`due-checks` gate, and passed — so the gate has now run in its real place, in both homes, not only in
+its own test suite.
+
+## 10. NOT yet validated
+
+**The gate has never fired on a real overdue date in the live register.** Every conviction shown here
+is from a temp-file fixture or a `FELHOM_GATE_TODAY` override. Its first genuine firing will be the
+next push on or after **2026-08-19**, when R-341's first check comes due. Until that happens, "it
+refuses the push" is proven in fixtures and *inferred* in production — the registration test proves it
+is wired into the runner, which is the part that could silently not be true.
+
+Also unvalidated: the block's own upkeep. Nothing checks that a row removed from the block was removed
+because the measurement was *taken* rather than because it was inconvenient.
+
+## 11. Teardown
+
+**This task provisioned nothing.** No VM, no container, no machine touched. The hub was read-only —
+snapshots of `hub.db` + `-wal` were taken into the session scratchpad for querying and are not
+committed.
+
+## 12. Register rows
+
+**The block, verbatim as committed:**
+
+```markdown
+<!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
+     One row per dated check. The R-number must have a row above. Dates are UTC.
+     Clearing a row means the check was DONE and its result recorded in that R-row —
+     or the date was deliberately moved, with the reason stated in the R-row.
+     This block is an INDEX, not the detail: the command and the preconditions live in
+     the R-row. Duplicating them here would create the second source this design avoids. -->
+| item | due (UTC) | what to measure |
+|---|---|---|
+| R-341 | 2026-08-19 | ep0 proxy fd count + ESTAB/CLOSE-WAIT split; PID must still be 551655 |
+| R-341 | 2026-08-25 | same, +7 d |
+<!-- DUE-CHECKS-END -->
+```
+
+**R-341's dates in the register matched the sheet exactly** — no disagreement to report.
+
+**R-343's verdict cell, verbatim:**
+
+> **OPEN — NEW 2026-08-18.** Deliberately NOT closed: the task's closing condition was *all five reads
+> clean, no directive served*, and read 5 shows a live box updated. It went cleanly and is the floor
+> working as designed — but a change recorded as a no-op when it moved a customer box is exactly the
+> kind of record that misleads later
+
+**R-342** filed **READY (S)**, owner *Viktor decides; CC executes*, quoting `stop2-snapshot.txt`
+verbatim on what the snapshot covers and does not.
+
+## 13. `unproven.py --summary`
+
+```
+where felhom stands — 55 claims, verified_on 2026-08-09
+  walked   23
+  partial  14   (6 cite evidence, 8 prose only)
+  built    14   (0 cite evidence, 14 prose only)
+  missing  4   (0 cite evidence, 4 prose only)
+  NOT WALKED: 32 of 55
+```
+
+**No number moved**, correctly: this task added a gate and three register facts, and walked no claim
+in the standing picture.
+
+## 14. Observations — noticed, deliberately not acted on
+
+- **`CLAUDE.md`'s gate list named only 6 of the 10 registered gates.** It was missing
+  `golden_currency_gate.py`, `wire_contract_gate.py` and `hub_copy_gate.py` — all registered weeks
+  ago. I completed the list rather than appending a 7th name to a list that was already wrong, since
+  the section's stated job is to name each gate. Effective line count 124 → 128 against a ceiling of
+  200, so no trim was needed.
+- **`documentation/architecture/00-capability-map.md` — no change, and this is the explicit
+  statement the task asked for.** No row's evidence citation names the floor or golden *version*:
+  line 153's publish-train row cites a runbook path, and line 44's golden literal is a dated
+  historical citation on the recovery-journey row.
+- **`drill-r50` will be dragged 0.213.0 → 0.216.0 by this floor if it is ever booted and reports.**
+  Its agent (0.129.0) meets `MinAgent`, so the floor would be served, not held. That is the floor
+  doing its job; noted so it is not read as a surprise later.
+- **The hub's log timestamps are CEST while its DB stores UTC.** Not a defect, but it makes a log
+  line and an events row for the same instant look two hours apart, which is worth knowing before
+  correlating them under pressure.
+- **`min_controller_version` and `artifact_*` live in the same `hub_settings` table but are saved by
+  different actions**, two hours apart today (11:00:59 vouch, 12:36:58 floor). That separation is
+  rule 2 working, and is why reading only one of them would give a misleading picture.

REPORT-spike-ep0-connections.md

@@ -0,0 +1,176 @@
+# REPORT — SPIKE: who holds ep0's connections open (2026-08-20)
+
+**Status: STOP 1 reached. Parts 0, 1 and 2 are complete. Phase C (Part 3) has NOT been run — it is your
+decision, below.** Everything done in this session was **read-only**. No machine was changed.
+
+---
+
+## ⛔ The decision waiting on you
+
+Phase C wants one overnight window on `demo-hp`. **The measurement found something that changes which
+mutation is worth running**, so there are two versions of it. Both are one mutation on a Tier-0
+disposable box, both arm a dead-man timer first, both are unattended.
+
+| | **Option A — stop `pvestatd`** (what the spike prompt specifies) | **Option B — stop `felhom-agent`** (what the evidence now points at) |
+|---|---|---|
+| what it tests | the original Q3: does the leak track the request rate? | does the leak track **our agent's** cycle? |
+| predicted result | **null** — leak unchanged at ~201.6/day, ~84 descriptors in 10 h, split evenly ~42/~42 | leak **halves** — ~101/day, ~42 descriptors in 10 h, split ~42 from `demo-felhom` and **~0** from `demo-hp` |
+| what it buys | **falsification.** If the leak *did* halve, the whole Part-1/2 attribution is wrong and must be withdrawn | **confirmation by a second, independent route.** A near-zero contribution from the quietened box is decisive |
+| cost if the attribution is right | a confirmed null — real evidence, but no new information beyond what Parts 1–2 already show | the sharpest possible confirmation |
+| risk | none beyond the window: `pvestatd` is PVE's stats daemon; the box keeps running, backups are not due until ~08-25 | slightly higher: the agent is our own product on the box. It would stop reporting to the hub for the window, and the hub's staleness watch may notice |
+
+**I would run Option A, tonight.** Two reasons. It is the mutation the prompt authorises, and standing
+rule 2 says exactly one mutation exists in this run — substituting one is my call to propose, not to
+make. More importantly, **A is the falsification test and B is the confirmation test**, and the
+attribution is already confirmed twice over (socket ownership on the boxes, and the access-log user
+agent, by completely independent routes). A test that can prove me wrong is worth more right now than a
+third test that can only agree with me.
+
+**What I need from you: "A tonight", "B tonight", "at the weekend", or "skip it".** If you pick B I will
+need you to say so explicitly, because it is a second mutation the prompt does not authorise.
+
+---
+
+## What I did, and what it found
+
+### Part 0 — the dated-check gate's first real conviction, captured before anything else
+
+The 2026-08-19 row was one day overdue. Every conviction this gate had produced before came from a
+fixture or a `FELHOM_GATE_TODAY` override; **this is its first firing on a real overdue date in the live
+register**, and it behaved correctly:
+
+- **exit code 1** — a verdict, not a crash (2) and not a pass (0);
+- the message **names R-341 and the days overdue** (`R-341  due 2026-08-19  1 day(s) OVERDUE`), so the
+  exit code is not doing the work alone — which is the failure mode this gate's own red-proof fell into
+  on 18 August;
+- inside `repo_gates.py` it is the **only** conviction: 9 gates OK, `CONVICTED: due-checks`.
+
+Captured verbatim in `part0-due-checks-gate.txt` and `part0-repo-gates.txt`. The row was **not** cleared
+to make the push work — it was cleared at Part 4, after the measurement existed and its result was
+recorded in R-341, which is the sanctioned order.
+
+### Q0 — R-341's first dated check: the slope is UNCHANGED
+
+Precondition passed: PID still **551655**, `NRestarts=0`, so the elapsed window is valid.
+
+**fd 17 → 405 over 166,251 s (46.18 h) = 201.6 fd/day**, Poisson 2σ 181.2–222.1.
+The prediction was pre-registered before the reading: **370–450**. **Observed 388.**
+Verdict: **unchanged**, the expected result, and not a failed upgrade.
+
+This settles the check on a window **88× longer** and a descriptor count **97× larger** than the
+30-minute windows the original answer rested on. Uncertainty drops from roughly ±50% to ±5%.
+
+**Composition:** ESTAB 0 → 388, and **CLOSE-WAIT is 0 — absent from the histogram entirely.** The
+incident document's original emphasis on `CLOSE-WAIT` is not merely the minority story; on this proxy
+generation that state does not occur at all.
+
+Runway to the 65536 ceiling: **~323 days (~2027-07-09)**.
+
+### Q1 — who is at the far end: exactly the two demo boxes, 194 each
+
+No third peer. Outcome (c) excluded. The identity is read from the API token name on every access-log
+line, not inferred from the address. `lsof` confirms the leak is sockets and nothing else: 390 of 405
+descriptors are TCP.
+
+**Persistence (31-minute diff of full 4-tuples): 388 in both readings, 0 closed, 4 new.** Not one socket
+closed. All carry keepalive timers with `retrans=0` — the far ends are answering, so these are not
+half-open sockets.
+
+### Q2 — outcome **(a)**, confirmed twice, exactly
+
+| instant | ep0 | `demo-felhom` | `demo-hp` | sum |
+|---|---|---|---|---|
+| 08:02:39 / 08:04:37Z | **388** | 194 | 194 | **388** |
+| 08:33:42 / 08:34:01Z | **392** | 196 | 196 | **392** |
+
+And the four sockets that appeared between the readings carry **the same four source ports** on ep0 and
+on the boxes. Both sides hold every connection.
+
+### The finding nobody predicted: the leak is ours
+
+`ss -tnp` on the boxes names the owner of **194 of 194** on each: **`felhom-agent`**, one PID per box.
+Zero are held by `pvestatd`. Zero by `proxmox-backup-client`.
+
+ep0's access log says the same thing by a completely independent route:
+
+| who | requests in the window | descriptors leaked |
+|---|---|---|
+| `libwww-perl` (pvestatd) | 81,192 | **0** |
+| `proxmox-backup-client` | 80,061 | **0** |
+| `Go-http-client` (**our agent**) | 811 (of which **387** `/snapshots` calls) | **388** |
+
+**One leaked socket per agent `/snapshots` call, within one.** 99.5% of the traffic produces 0% of the
+leak.
+
+**Mechanism, named from source** — `felhom-agent/internal/pbs/client.go:56-60` builds
+`&http.Transport{TLSClientConfig: tlsCfg}` as a composite literal, so `IdleConnTimeout` is the zero value
+(= no limit; `http.DefaultTransport` sets 90 s and a literal does not inherit it), and
+`cmd/felhom-agent/main.go:1486` builds **a fresh client every cycle**, as its own doc comment states.
+`CloseIdleConnections`, `IdleConnTimeout` and `MaxIdleConns` appear **nowhere in the agent repo**.
+Cadences reconcile without fitting: 900 s hub poll (184.7 cycles) + 6 h verify cadence (7.7) = 192.4
+predicted against **194 observed per box**.
+
+**No fix is proposed** — the spike-first gate forbids it, and the spec is a separate task. Filed as
+**R-344**, with the open design questions listed rather than pre-answered.
+
+### Control window: clean
+
+No reboots (ep0 up 17 d, boxes up 10 d), no daemon restarts (`NRestarts=0`), no agent restarts, **no
+request-rate gap** (3,536–3,553 per hour, every hour), **no HTTP errors at all** (every response 200
+except 8 expected 101s), no tunnel flap. **No backup ran inside the window** — the last offsite protocol
+upgrades were 08-18 03:57/03:58Z, *before* `t0`, and the tier is weekly with the next run due ~08-25, so
+**tonight's window is also clear of one.** Routine verify jobs and restore-test reads did occur; they are
+1.2% of traffic and are already counted inside the 194/box reconciliation.
+
+**Hub:** no `*_unreachable` or `*_recovered` event; the PBS-DR gauge refreshes on schedule and host
+reports land from both boxes. **Honest limit:** the hub pod is 39 h old, so hub-side logs cover 36.6 of
+the window's 46.2 hours; the first 9.6 h rests on ep0's own evidence.
+
+---
+
+## Register changes
+
+- **R-341** — first check recorded (taken at **+46.2 h, not +24 h**; the delay was pure elapsed time and
+  the longer window is stated as a **better** measurement, not a degraded one). **2026-08-19 row removed**
+  from `DUE-CHECKS`; 2026-08-25 kept, with the Phase-C perturbation quantified against it (**~3%** shift
+  on the 7-day slope even under the hypothesis we expect to be false — the reading stays usable).
+- **R-336** — mechanism named; **premise corrected and re-ranked**. Its recorded next step would have
+  produced a null result and read as a failed fix. The poll rate is now a scaling/cost item; the leak fix
+  is R-344. **Q3's proportionality verdict is explicitly NOT recorded** — Phase C has not run.
+- **R-340** — noted which of its wanted observations this run already produced, so the health-op task
+  reuses them rather than measuring a protected machine a third time. Only the loopback probe is still
+  owed.
+- **R-344 (new)** — the agent's per-cycle transport leak. READY (S).
+- **R-345 (new)** — `hub/Makefile` lines 21–22 tag and push `:latest`, which two rule files forbid.
+  READY (XS).
+- **R-346 (new)** — `ActiveEnterTimestamp` reads 5 h 56 m early for this proxy generation (the upgrade
+  re-exec'd rather than restarted, so `NRestarts` is still 0). Anchoring a slope on it gives ~12% low.
+  READY (XS).
+
+## Deliverables
+
+- `documentation/audits/SPIKE-ep0-established-connections-2026-08-20.md` — the findings.
+- `documentation/audits/evidence-ep0-established-connections-2026-08-20/` — 11 raw evidence files,
+  including the **pre-registered Phase C prediction**, committed before the mutation exists.
+- `documentation/backlog/OPEN-ITEMS.md`, `STATUS.md` — as above.
+
+## CI, checked by run ID
+
+**id `360` / run_number `237`, `head_sha 19672e685`, conclusion `success`**, started 2026-08-20 08:41:50Z
+— this run's own push. The four runs before it (ids 356–359) are also `success`, so the "green since run
+356" baseline in the spike prompt holds and nothing was inherited red.
+
+*Numbering note:* the API exposes two numbers per run and they differ by 123 here. The prompt's "run 356"
+matches the **`id`**, not the `run_number`; both are recorded in `evidence-…/ci-run-by-id.txt` so the
+reference is unambiguous.
+
+The pre-push hook ran `repo_gates.py --fast` and reported **all 10 gates OK** before the push proceeded —
+including `due-checks`, which convicted at Part 0 and passes now that the row is properly cleared. **No
+`--no-verify` was used**, and none was needed.
+
+## What is inconclusive
+
+**Q3 is unmeasured**, by design. Everything stated about proportionality is a labelled prediction. Also
+unexplained: why the two boxes' leaked counts are *exactly* equal at two separate instants rather than
+merely close. And whether restore-test reader connections leak too was not separated out (≤2% of the
+total, inside the noise).

REPORT.md

@@ -1,169 +1,181 @@
-# REPORT — The door, part one: a correct code stops being called wrong (2026-08-12 night)
+# REPORT — SPIKE R-459: is a skipped MariaDB upgrade harmless, and what does fixing it cost? (2026-09-06)
 
-**Three repos.** hub **v0.103.0** · agent **v0.129.0** · controller **v0.214.0** · golden **0.214.0**.
-Register: **R-311 CLOSED**, R-307 CLOSED, **R-312 / R-313 / R-314 / R-315 opened**. Ceiling
-R-310 → **R-315**.
+*Overwritten each session. Nothing durable lives only here.*
 
----
+> **OUTCOME A, qualified — and the trade this task was commissioned to price DOES NOT EXIST.**
+> The skipped conversion is **stable but never self-resolving**. Fixing it costs **7 seconds** and
+> does **not** cost the ability to abort, which is what B assumed. It is not C either: the conversion
+> **succeeded** across the multi-major jump.
 
-## 1. The spike's answer, first and in plain language
+## 1. Confirmed baselines — none had moved
 
-**Can a customer restore from a set-aside store with the machinery that already exists? NO — and
-building it is new surface, not wiring.** Established read-only, at `file:line`, before a line was
-written.
-
-Every restore entry point resolves the repository from `settings.GetOffboxTarget()` and the password
-from the single `offboxPwPath()` file: `offboxLatestSnapshot` (`offbox_restore.go:85-86`),
-`offboxSnapshotSize` (`:139-140`), `RestoreOffboxScratch` (`:206+`). **A grep for a repo-path
-parameter anywhere in the restore chain returns nothing.** The only seam that installs a recovered
-password, `InjectOffboxPassword` (`offbox.go:667`), writes that same one file — i.e. **adoption**.
-
-What the drill did to read the set-aside store was `restic` **by hand**, with `-r <alt repo>` and an
-overridden `RESTIC_PASSWORD_FILE`. **That distance is exactly what (b)-to-(c) costs.**
-
-**So the session halted at Part 3 by its own rule, shipped Part 2, and hands back options → R-312.**
-Cost to find out: ~35 minutes, read-only.
-
-## 2. Part 0 — the countdown, cancelled on your ruling
-
-Through the product's own operator path (`--abandon-stop`, which refuses rather than silently
-no-opping), with the container **stopped first** so the running controller could not overwrite
-`settings.json` from memory. **Proved, not trusted to the exit code:**
-
-- `abandon_started_at` and `abandon_at` — **gone**. `AbandonStatus` returns `Active=false` when
-  `AbandonAt` is empty (`offbox_abandon.go:111-113`), so no countdown renders.
-- `abandon_repo_path` — **deliberately kept**, as the pointer to the preserved store.
-- The set-aside store — **still there**: 36 snapshot objects, full `config/data/index/keys/locks/
-  snapshots` structure. Both repositories still on the endpoint. **Nothing deleted anywhere.**
-
-**And the thing you should know about what was preserved (R-313):** it holds **36 snapshots and one
-key slot**, and it does **not** open with the box's current password (`Fatal: wrong password or no key
-found`, exit 1 — measured). Its key is the one hashed `48741892f0ef…` — retained row id 4,
-`identity_blob` **NULL**, a pre-v0.93.0 row. **The material was dropped by the R-198 defect during its
-two-month window, so no recovery code in existence opens that store.** Keeping it is still the right
-call — deleting is irreversible and a decision, not an accumulation — but it is 36 unreadable
-snapshots, and that is the concrete, still-present cost of R-198 sitting on the endpoint.
-
-## 3. Part 2 — what shipped, and a correction to the premise
-
-**The premise needed correcting first.** The task described the customer being told *"the recovery code
-did not open the sealed bundle"*. That is the **agent's local-API** reply. The **customer-facing
-screen already hedged** (R-222/R-226) — it named both causes, named the kept package and its date, and
-said it could not tell them apart. That was **honest**; it could not tell them apart **because nothing
-ever looked**. So what shipped is smaller and more precise than "stop the lie": **the hedge becomes an
-answer.**
-
-- **hub v0.103.0** — `GET /hosts/<id>/escrow/retained`, the **first production caller
-  `ListSupersededEscrow` has ever had**. Self-scoped identically, same recovery-mode gate, same audit
-  event written before the bytes leave, capped at 16. Rows with a NULL `identity_blob` are **withheld
-  and counted** (`unopenable_count`): they can never open what the caller is asking about, and serving
-  them would let the screen promise recovery on exactly the boxes the original defect hurt.
-- **agent v0.129.0** — retained packages tried **only after** the current one refuses; `422` with
-  `superseded_at`; bounded at 6 attempts (~1 s of scrypt each); fail-safe in every direction.
-- **controller v0.214.0** — class `RecoveryCodeOpensRetained`, gated on MinAgent **0.129.0** via a
-  **second, separate** trust flag (a box can sit between 0.126.0 and 0.129.0). The message says the
-  code is correct, names the date, says the package is kept, says the **current** backups are
-  unaffected, and **promises no restore** — it routes to support, which can do it.
-
-**The trade you should see stated:** the hub still cannot read any of it — sealed bytes in, sealed
-bytes out, no decrypt path, no recovery code ever held. What widens is **volume**: a host key that
-could fetch one opaque package can now fetch N, bounded by self-scope, the recovery-mode gate and the
-cap.
-
-## 4. Red-proofs — and where the lie actually lives
-
-Every mutation asserted to have applied before its run.
-
-| Repo | Mutation | Outcome |
+| repo | task's baseline | found |
 |---|---|---|
-| hub | serve the CURRENT row instead of retained | FAILS (count 2→1) |
-| hub | drop the unopenable guard | FAILS (count 1→2, unopenable 1→0) |
-| hub | drop self-scope | FAILS (403→200) |
-| hub | collapse the route suffix | FAILS (count 1→0) |
-| **agent** | **remove the retained lookup** | **FAILS — the fail-closed wrong-code error returns. THE LIE COMES BACK.** |
-| agent | + 6 more (nil fetcher, wrong code, fetch failure, bounded attempts, predates-field, success path) | all pinned |
-| controller | delete the new case | FAILS — but the customer gets the **neutral** message, because R-224's safe default catches it |
-| controller | make 422 unconditional | FAILS — an agent that never looked is read as having looked |
-| controller | route 400 to the new class | FAILS — a mistype is congratulated |
+| app-catalog-felhom.eu | `0474ce387e6f` | `0474ce387e6f` |
+| felhom.eu | `a1a6c73fe132` | `a1a6c73fe132` |
+| felhom-controller | `bab82c471e03` | **not touched** |
 
-**Answering the question directly:** the lie returns when the **agent's** retained lookup is removed,
-not when the controller's case is. R-224's safe default is doing its job one layer up.
+Highest `R-` id **462**, confirmed. Minted **R-463**, **R-464**. No version bump, no release.
+**The task's §2 table was verified and is exactly right**: four MariaDB apps at the stated pins, and
+**no `MARIADB_*` env in any of the 53 templates.**
 
-## 5. The claim guard, and a gate whose positive control failed
+## 2. The outcome, and the evidence that assigns it
 
-**The claim guard had a blind spot the size of the recovery screen** — it scanned templates only,
-while every recovery message is a Go string in a handler. It now scans `recovery_handlers.go` too, and
-**on its first run convicted a pre-existing unregistered claim**. 8 → 10 registered claims.
+| | | |
+|---|---|---|
+| **A** | unconverted datadir is benign | **THIS ONE, qualified** — 5 of 5 restarts, no degradation. But the engine says a check is required *every* start, so "benign" holds only as measured: no decay over restarts, one seeded record, over minutes. |
+| **B** | converting works, the abort dies with it | **NO.** Converting works; the abort still starts and serves. |
+| **C** | converting fails, the jump is too big | **NO.** It succeeded in 7 s and took its own backup first. |
 
-**The wire-contract gate: declared, and honestly weaker than it looks (R-315).** The hub response was
-made a **named type** so the gate could resolve it; the wire is declared as a fourth ROOT and the tag
-count rose **174 → 182**, so the fields are inspected. But a positive control — renaming the
-agent-side `superseded_at` tag — **still passed**, because the check is repo-wide name-presence and
-the string also occurs as a map key elsewhere. The gate documents this ("name-reachability is not
-use"), so it is a known limit, not a regression — **but declaring this wire bought documentation, not
-enforcement**, and saying otherwise would have been false.
+## 3. Observable 1 — `mariadb_upgrade_info`, all four moments, verbatim
 
-## 6. Live state
+| moment | state | contents |
+|---|---|---|
+| 1 | fresh 11.6 datadir | `11.6.2-MariaDB` (14 bytes) |
+| 2 | 12.3 started on it, skip logged | **`11.6.2-MariaDB` — unchanged** |
+| 3 | after 5 restarts of 12.3 | **`11.6.2-MariaDB` — still unchanged** |
+| 4 | comparison arm, after conversion | **`12.3.3-MariaDB`** |
 
-| | |
+The engine serving at moments 2–3 was `12.3.3-MariaDB`, confirmed by `mariadb --version` — so the
+mismatch is real and not the wrong container.
+
+## 4. Observable 2 — the engine's own verdict, invocation, output and exit code
+
+Check mode located in the image, not assumed: **`--check-if-upgrade-is-needed`**.
+
+```
+$ docker exec bookstack-db sh -c 'mariadb-upgrade --check-if-upgrade-is-needed \
+      --user=root --password=$MYSQL_ROOT_PASSWORD'
+Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!
+[exit=0]
+```
+
+After conversion:
+
+```
+This installation of MariaDB is already upgraded to 12.3.3-MariaDB.
+There is no need to run mariadb-upgrade again.
+[exit=1]
+```
+
+**The two runs establish the exit-code semantics between them — 0 = needed, 1 = not.** Measured, not
+read from documentation, and worth stating because the polarity is the reverse of the usual
+convention.
+
+**A false start, recorded because it nearly produced the wrong answer:** without credentials the same
+command returns `ERROR 1045 (28000): Access denied … FATAL ERROR: Upgrade failed` with **exit 1** —
+an authentication failure wearing the shape of a verdict. → **R-464.**
+
+## 5. Observable 3 — the restart series: 5 of 5
+
+| restart | settled | readback | `upgrade_info` | engine check | entrypoint |
+|---|---|---|---|---|---|
+| 1–5 of 5 | yes, each | **pass, each** | `11.6.2-MariaDB`, unchanged | `Check required!`, each | `[Note]`, never escalated |
+
+**It does not degrade. It also never heals.** R-459's hypothesis that the Note might become a Warning
+or an Error is **not supported**.
+
+## 6. Observable 4 — the comparison arm
+
+Scratch copy of the template **inside the guest** with `MARIADB_AUTO_UPGRADE=1`. **Nothing with
+`MARIADB_` in it was committed to the catalog.**
+
+**The conversion succeeds**, verbatim and in order:
+
+```
+[Note] [Entrypoint]: Starting temporary server
+[Note] [Entrypoint]: Backing up system database to system_mysql_backup_11.6.2-MariaDB.sql.zst
+[Note] [Entrypoint]: Backing up complete
+[Note] [Entrypoint]: Starting mariadb-upgrade
+[Note] [Entrypoint]: Finished mariadb-upgrade
+```
+
+**Cost:** `mariadb-upgrade` itself **17:33:24 → 17:33:31 = 7 s**; whole TO step **35.9 s** with it
+against **~10.5 s** without → **≈ 25 s of extra startup, once**. **Measured on a nearly empty
+database** — it works over system tables rather than row data, so it should scale with table count,
+**but this run did not measure that** and 7 s must not be quoted as a fleet figure.
+
+### The decision-relevant fact — §14.6 of the task
+
+**Converting does NOT kill the abort.** With the datadir at `12.3.3-MariaDB`, putting **11.6** back:
+
+```
+abort: starts-and-serves        settled 10.5 s        readback: PASS
+survived a further restart:     Up 25 seconds (healthy),  upgrade_info: 12.3.3-MariaDB
+```
+
+**…but the engine calls that state unsupported, and the entrypoint hides it.** → **R-464**:
+
+| asked | answer |
 |---|---|
-| controller | **0.214.0** on guest 9201, `Up … (healthy)` |
-| agent | **0.129.0** on `felhom-pve`, unit active, journal clean |
-| hub | **0.103.0** — see §7 |
-| golden | **0.214.0** baked + published |
+| the entrypoint, every start | `[Note] [Entrypoint]: MariaDB upgrade not required` |
+| `mariadb-upgrade --check-if-upgrade-is-needed` | `FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!` |
 
-### Live proof on hardware — the 422, end to end
+**So "the abort works" is an observation that it started and served — not a claim the datadir is
+sound.** One restart cycle, one seeded record. Saying more would repeat the mistake this task exists
+to correct.
 
-```
-OLD code (opens retained row 11)   HTTP 422  opens_retained: True
-                                             superseded_at: 2026-08-12T15:18:55Z
-                                             retained_has_restic_pw: True
-                                             "the recovery code is correct, but it belongs to an
-                                              EARLIER sealed package (superseded …), not the one
-                                              currently held"
-WRONG code (negative control)      HTTP 400  "the recovery code did not open the sealed bundle"
+## 7. The harness change, and E3b re-run showing it
+
+`upgrade-test.py` gained **`engine_state_after`**, reported **beside** the verdict and never folded
+into it — an unconverted datadir is not *known* to be a failure, so a verdict that said so would
+encode an unproven judgement. E3b re-run:
+
+```json
+"verdict": "proven",
+"engine_state_after": {"bookstack-db": {
+  "image": "mariadb:12.3",
+  "answer": "11.6.2-MariaDB| Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required! [exit=0]"}}
 ```
 
-The hub half measured directly too: `GET …/escrow/retained` → **200**, `count=2`,
-**`unopenable_count=1`** — that one being retained row id 4, the pre-v0.93.0 row whose material R-198
-destroyed. The withholding rule is doing exactly what it was written for, on real data.
+**The exact thing the harness was blind to, now visible next to a green verdict.** A PostgreSQL probe
+is included; it has **never run against a real Postgres major** (→ R-463).
 
-**What was NOT walked, and why.** The customer's rendered sentence was **not** produced end-to-end.
-`recoveryUnlockHandler` redirects to `/backups/remote` when `!recoveryOffer()`, and `demo-felhom`
-holds its own repository password again (restored yesterday), so it is correctly **not** in the
-offered state. Walking it would mean removing that password to fake a rebuilt box — destabilising a
-healthy machine to render a sentence whose logic is pinned by six handler tests and whose upstream 422
-is proven live. I did not. **Method stated: endpoint-level for the agent and hub, handler-level for the
-message.** What the customer DOES see on this box today is the orphan card, and it is honest:
-*„Megnyitni innen egyelőre nem lehet, és ez nem a kódodon múlik."*
+## 8. Register — 206 open before, 208 after; closed 168, unchanged
 
-### A correction I have to make about my own last report — R-308 was wrong
+| row | disposition |
+|---|---|
+| **R-459** | **NARROWED, P2 → P3, WAITING-ON-OPERATOR.** Consequence measured; the expected trade does not exist. The fleet-wide env change remains the operator's call (4 apps). *VIKTOR rules, CC implements* |
+| **R-463** | **OPENED**, P2-MEDIUM, CC — the PostgreSQL analogue. **11 templates, 8 on `postgres:16-alpine`**; register grep for `pg_upgrade`/"postgres major" returned **0**, confirmed twice. **Deliberately not measured here.** The two engines fail in **opposite** directions: MariaDB skips quietly, Postgres **refuses to start** — so this one cannot hide, it presents as eight apps down at once |
+| **R-464** | **OPENED**, P3-LOW, CC — `MariaDB upgrade not required` is printed on an unsupported downgrade, so that line cannot be a soundness signal. Same class as "presence is not success" and R-443's HTTP 200 over a crash-looping app |
 
-I reported that the stored controller password no longer opens `demo-felhom`. **It does.** I had
-stripped only DOUBLE quotes from the `~/.config/credentials` value; the values are wrapped in
-**SINGLE** quotes, so I was sending a literal `'` as part of the password. Unquoted correctly it is 13
-characters and logs in first try — **HTTP 302 with a session cookie**.
+## 9. Teardown — three layers
 
-The same bug then made this session's first live R-311 test read as a **failure** (HTTP 400) for
-twenty minutes, and I nearly filed the fix as broken. It is the **third** wrong "the credential is
-stale" verdict this project has produced from that one trap. R-308 is **withdrawn**; the real lesson
-is filed with it — never let a shell decide what a secret is.
+| layer | result |
+|---|---|
+| **1 — machine** | guest **9402 destroyed**; `pct list` shows only 9201. `scratch-r459` **removed**. Downloaded template deleted. |
+| **2 — host** | **`local-lvm` 30.53 % before and after — never touched.** `local` 19 630 088 → 19 631 740 KiB (+1.6 MB). `pct fstrim 9402`: **35.9 GiB trimmed**. Guest peak 3.2 GB / 2.10 GB images. |
+| **3 — hub** | **Checked, not asserted:** `/hosts` = exactly `demo-felhom-8363b5`, `demo-hp-bb76ea`; **0 customers**. **This run created no customer, no appliance, no host record.** |
 
-## 7. What was dropped, named plainly
+Evidence off after **each arm**, before teardown: `documentation/audits/r459-spike-2026-09-06/`
+(20 files, 164 KB). Scanned for secrets before commit: clean.
 
-- **Part 3 (the route) — HALTED at the spike, by the task's own rule.** → R-312.
-- **§7's fixture walk was not re-run end-to-end.** Yesterday's drill already proved the byte-identical
-  restore from a set-aside store; today's change is upstream of it (which sentence is shown), the
-  dashboard is unreachable headlessly (R-308), and the restore route does not exist (R-312). What was
-  proved live is the 422 itself.
-- Explicitly out of scope and still open, so it does not read as forgotten: **R-305** (the removal fix
-  helps a machine once — the tester's second reinstall still hits it), the hub emails naming the
-  retired secret, **R-309** (the runbook's publication claim), the CI runs that fail with no log, the
-  twenty unread facts, the nine grey claims, **R-303**.
+## 10. Claims in the task that turned out to be wrong, named
 
-## 8. Bypass, stated as required
+1. **§11's venue is still stale, exactly as R-461 records. `/mnt/nvme-1tb` does not exist** — the NVMe
+   is at `/mnt/hdd_1`. The scratch storage went at *its* root, honouring the rule's reason; `local-lvm`
+   read 30.53 % before and after. **`drill-r50` (VM 300) still does not exist** — `qm list` returns
+   nothing. Both were already filed as R-461 yesterday; this run is the second session to work around
+   them, which is the cost that row predicts.
+2. **§6 Observable 2 assumed a check mode would exist but told me not to assume its flag — correct,
+   and the caution earned its place.** The flag is `--check-if-upgrade-is-needed`, and **its exit-code
+   polarity is the reverse of the usual convention** (0 = work needed). Assuming either the name or the
+   polarity would have inverted the headline.
+3. **§7's Outcome B was the expected result and it is FALSE.** The abort survives a real conversion.
+   The task was right to name three outcomes and to say "do not steer" — the run steered nowhere and
+   landed outside the shape the brief most anticipated.
+4. **§9's Postgres numbers were exact**: 11 templates, 8 on `postgres:16-alpine`, register grep 0.
+5. **§4.7's pointer was right** — `mariadb_upgrade_info` was already visible in the persistence-sweep
+   probe, and it is the observable that carried this task.
 
-`git push --no-verify` was used **once**, on `felhom-agent`. The `release-complete` gate refuses a
-CHANGELOG entry whose tag and package do not exist; `release-agent.sh` refuses a tree that is not
-pushed. Circular by construction. The bypass was immediately followed by the real release
-(`release-agent.sh 0.129.0`), and the gates were re-run afterwards: **green**.
+## 11. Observations — noticed, documented, NOT acted on
+
+1. **`mariadb-upgrade` without credentials returns a confident-looking failure that is an auth error.**
+   **FILED: R-464**, which carries it as the second half of the row.
+2. **The conversion leaves its own backup in the datadir** (`system_mysql_backup_*.sql.zst`, 622 437
+   bytes here), which any volume-level backup will then copy. **NOT-A-FINDING: it is upstream's
+   deliberate safety net and it is harmless; recording it in the audit is enough to stop the next
+   person reporting an unexplained file.**
+3. **Python heredocs do not survive three shells (`ssh` → `pct exec` → `bash`).** Two scripts had to be
+   written locally and pushed as files. **NOT-A-FINDING: it is a working technique, not a property of
+   the product, and it is now written into the audit where the next session will meet it.**

REUSE.md

@@ -65,6 +65,7 @@
 | `(*Server).hostStatus` + `hostStatusClass`/`hostStatusLabel` | hub/internal/web/hosts.go (~L16/34/48) | `(lastReport *time.Time) string` | Host liveness badge | Uses the SAME threshold as HostStalenessChecker (down = 2× stale) — never invent a second definition. |
 | `parseSQLiteTime` | hub/internal/store/store.go (~L1160) | `(s string) time.Time` | Parsing ANY timestamp read from SQLite | modernc/sqlite returns multiple formats; raw `time.Parse` will intermittently zero out. Always use this. |
 | `compareVersions` | hub/internal/web/server.go (~L571) | `(a, b string) int` | X.Y.Z comparisons in web (floor checks, update-available) | Returns 0 on parse error — unparseable compares as "equal" (see §3). |
+| `reportBackupCard` + `backupCardView` + `fmtBytesAuto` (R-331, v0.109.0) | hub/internal/web/backup_card.go | `(reportJSON string) backupCardView` / `(int64) string` | THE customer-page Backup card — everything it shows about a customer's backups | **Reads the report's `offsite` object, NEVER `backup`.** The `backup` object's `snapshot_count`/`repo_size_mb`/`integrity_ok` have had no producer since slice 8C and rendering them told every operator every customer had `Snapshots 0` (measured on demo-hp over a repo holding 67). **Always consult `StatsKnown` before believing a zero** — absent/false means "never measured", NOT "empty", and those are opposite news (R-225 measured the same confusion one layer down). Resolved in Go, not the template, because a `{{if}}` chain over `Report`'s `map[string]interface{}` float64s cannot keep the absent/zero distinction the card is entirely about. `fmtBytesAuto` scales MB/GB/TB — do NOT swap in `fmtBytesGB`, which renders demo-hp's real 140 829 678 B as `0.1 GB`. |
 
 ### Host views & lifecycle / offsite endpoints (v0.47.0, hub/internal/web + store)
 
@@ -174,6 +175,7 @@
 | Inline `stringData` secrets à la manifests/felhom.secret.yaml | Commits real credentials to git (healthchecks superuser pw, umami APP_SECRET/POSTGRES_PASSWORD, gitea-creds admin password still live there). | Out-of-band `kubectl create secret` + `secretKeyRef` (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md) |
 | `kubectl apply` / `kubectl set image` on manifests/ | ArgoCD app `felhom` reverts drift on next sync; live state lies about git. | Edit manifest in git → push → ArgoCD sync (CLAUDE.md steps 3–5) |
 | `:latest` image tag in manifests | Re-push doesn't change the manifest → no redeploy; Synced/Rollback misreport. | Pinned version tag, bumped per deploy |
+| The report's `backup` object for snapshots / repo size / integrity (`snapshot_count`, `repo_size_mb`, `integrity_ok`) | **No producer since slice 8C** — the controller's `buildBackupReport` leaves all four zero deliberately and says so. Rendering them gave every customer `Snapshots 0 · Repo Size 0 MB · Integrity Unknown` indefinitely (R-331, measured on demo-hp over a repository holding 67 snapshots). `integrity_ok` is worse than stale: the controller runs no integrity check at all, so it can only ever be "Unknown" or a lie. | The report's `offsite` object (`backup.OffboxReportStatus`) via `reportBackupCard` — and check `stats_known` before trusting a zero |
 | grep/regex hunting emoji in website HTML | Windows grep false-negatives multibyte emoji (proven in D0). | `python scripts/site_gates.py` (codepoint-range check) |
 | Adding a website page without touching site_gates.py | `PAGES` list (scripts/site_gates.py ~L22) is explicit — an unlisted page is silently ungated (BOM/nav/emoji drift undetected). | Add the filename to `PAGES` in the same commit |
 

STATUS.md

@@ -1,6 +1,38 @@
 # STATUS — what works, what's broken, what's next
 
-**Updated 2026-08-12 (night — the door, part one).**
+**Updated 2026-09-13 — "delete my data too" now deletes the data, or tells you it could not. Until today the box said it worked and left everything on the drive. Live on both machines (0.236.0), proven on the HP with a throwaway Nextcloud. Nothing needs you. Item 7 is closed: you decided it on 2026-09-02 and the page still listed it open.**
+
+**Updated 2026-09-06 (third pass) — I chased down the BookStack database problem I found this
+morning. Good news: it does not get worse, and fixing it costs seven seconds and loses us nothing.
+The catch I expected — that fixing it would stop us being able to go back — turned out not to be
+real. TWO THINGS NEED YOU: item 11 (how wide to take the upgrade testing) and item 12 (one small
+yes/no on the database setting).**
+
+**Earlier 2026-09-06 (second pass) — I built a machine that upgrades a real app with real data in it
+and then asks the app whether the data is still there. Three apps, five real upgrades: the data
+survived every time. It also found a genuine problem in our own BookStack setup.**
+
+**Earlier 2026-09-06 — a restart no longer changes which version an app runs. Fixes still arrive
+every 15 minutes, and a broken app definition still repairs itself. Only the Update button moves a
+version now. Live on the HP (0.235.0). Two old items closed: the Hetzner e-mails are answered, and
+the Docker Hub login is in place.**
+
+**Earlier 2026-09-03 — you spotted that OpenGist had no label. You were right, and it was a real gap:
+the label only appeared on apps something had restarted. Fixed and live (0.234.0). Every app on both
+machines now carries one.**
+
+**Earlier 2026-09-02 — the box now writes down which version of each app it is running, and shows one
+small label saying whether it is up to date: „Naprakész" or „Frissítés elérhető — 52 napja". No version
+numbers, and nothing about updating changed.**
+
+**Earlier 2026-09-01 (third pass) — the backup work is FINISHED for beta, and I have written down
+where it stops. One alarm that was telling you something untrue is fixed and live (hub 0.111.1).
+ONE THING is waiting on you: two short e-mails to Hetzner, drafted and ready to paste.**
+
+**Earlier 2026-08-31 — I measured whether the box could test its own remote restore without you,
+then built the narrow version you picked. The measurement is why it is 3 seconds a night and
+not an evening's work.**
+
 
 > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
 > part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.**
@@ -8,98 +40,426 @@
 
 ## Waiting on you
 
-*Three decisions. Each says what it would cost to leave alone, because two of them quietly choose an
-outcome if you do not answer. The register row is the detail, not the decision.*
+*This section is allowed to be longer than one screen, and each item says what happens if you do
+nothing.*
 
-### Should a customer be able to get their old backups back themselves, or is that a phone call?
+1. **Two things are waiting on you: item 11 (how wide to take the upgrade testing) and item 12 (a small yes/no about the database setting — I have measured both costs).** Item 4 (the Hetzner e-mails) is answered and is being handled in a separate session. Item 7 — the safety-copy decision — was ruled on 2026-09-02 and is now marked closed below (it was **not urgent any more**: the thing that made it urgent was that a restart could upgrade an app behind your back, and as of today it cannot. Item 10 is new and needs nothing from you. Item 5's alarm mail can now be ignored for good. Otherwise: Both problems the overnight test found are fixed and proven on
+   the real machines:
+   - the background job that could delete a live restore's lock now waits its turn — and the check
+     that finds the next one like it is a test, not a comment, so it cannot come back quietly;
+   - the nightly backup check now runs on `demo-felhom`. It said **pass** there this morning, on a
+     machine where it could not start at all yesterday.
+   The golden carrying both is baked, vouched, and the floor is raised. `demo-felhom` picked it up
+   **by itself in about 20 seconds**.
 
-Since Tuesday a machine recognises an older recovery code and says so honestly, but it cannot hand the
-files over — it tells the person to write to us, and we can do it by hand.
+2. **Whether to keep the test app `bentopdf` on `demo-hp`.** I deployed it last night because it
+   is the ONLY app of our 53 with neither a database nor stored files — which makes it the only
+   way to prove the new check does not cry wolf on an app that legitimately has nothing. It
+   passed silently, which is what we needed to see. **My pick: keep it**, as a permanent control.
+   **If you do nothing:** it stays, using almost no space. Say the word and I remove it.
 
-- **Build it:** the restore code has to accept a second location and password instead of only its own.
-  Contained — three functions and a screen — plus one genuine design question: what a customer sees
-  when they have several old sets and must pick one.
-- **Leave it:** nothing breaks. Every customer in this position becomes a support conversation, and we
-  keep a promise we can only keep manually.
+3. **Nothing else about this release.** Everything in 0.232.0 ships in the controller image plus
+   two register lines in the hub (already live). No customer action, no data migration, no
+   credential change.
 
-**If you do nothing:** the honest message stays and the work never gets scheduled. Nobody is blocked;
-this is the one decision here with no deadline of any kind. *(register: R-312)*
+4. **~~Please send two short e-mails to Hetzner.~~ DONE — you sent them and Hetzner replied (2026-09-06). The reply is being worked in a separate session; nothing about it belongs to the update work.** The background below is kept because it is why the questions were asked.
+   `felhom.eu/documentation/runbooks/provider-questions-2026-09-01.md` — open it, copy, send. No
+   password or key is in that file, and none should be added.
 
-### The old copy on the demo machine cannot be opened by anyone. Keep paying to store it, or delete it?
+   **Why.** Yesterday I told you the snapshots make a wiped backup survivable: lose about a day, copy
+   the rest back file by file. **The first half is still true. The second half is not, and I found
+   that out by trying it.** I tried **777,600** snapshot names on the storage, over nine days, in
+   Hetzner's own naming style. **None of them opened.** Then I found why: your data and the snapshot
+   door sit on **two different drives** inside the storage, and the door for your data **does not
+   exist at all**. So there is no way in from the machines.
 
-You told me to keep it, and I did. Then I found out what it is: 36 backups and a single key, and that
-key was destroyed by the bug we fixed on 4 August. **No recovery code in existence opens it.**
+   **What is still true, and it matters:** a machine that wipes its own backup **still cannot touch
+   the snapshots of it**. The older copy is there. What we do not have is a way to reach it.
 
-- **Keep it:** pennies of storage, and it stays as the one physical example of what that bug cost.
-- **Delete it:** irreversible, and the example goes with it.
+   **The two questions.** One: can the **main** account pull single files out of a snapshot? Two: on
+   one of Hetzner's own tools, is a "cannot delete" switch forced by them, or chosen by the machine?
+   **The second one could remove the whole problem** — no new hardware, no moving anyone's data.
 
-**If you do nothing:** it stays forever and stops being a decision — which is how five scratch
-customers accumulated. Nobody is blocked. *(register: R-313)*
+   **If you do nothing:** we cannot finish this. The backups keep working and keep being checked; we
+   simply cannot say what a wiped backup costs, and I would then put this risk back near the top of
+   your list. **My pick: send both. It is five minutes and it decides an evening's work.**
 
-### When a machine is in two kinds of trouble at once, should it say both things?
+5. **You will have received an alarm email from me today about `demo-hp` losing 65 backups. It is a
+   test and nothing is wrong.** I built the new "someone deleted the backups" alarm and had to fire it
+   once for real to prove it reaches you. Subject: `[Felhom] 🔴 demo-hp: offsite_snapshots_dropped`.
+   **No backups were deleted.** **If you do nothing:** nothing — but please do not act on that one
+   mail. **Closed now:** that was the only such mail, it was a test, and the alarm's wording has since
+   been corrected (item under *Decided* below). Nothing further is needed from you here.
 
-A machine can count down to deleting its old backups while also reporting that it cannot open its new
-ones. Both cards are true; together they are bewildering.
+6. **Whether to change the hub password** (R-350). I printed it into my own session log on 20 August.
+   Not in git, not in any saved file — in the log on this machine. **If you do nothing:** it stays as
+   it is, at the risk you accept by leaving it. I can change it without ever showing you the new one.
 
-- **Leave it:** two true statements, confusing side by side. Nobody has been hurt by it.
-- **Hide the older card during a countdown:** tidier, and it risks hiding a real second failure — which
-  is why I have not done it.
+7. **CLOSED 2026-09-13 — you decided this on 2026-09-02, and the page kept listing it open.** The ruling is recorded in `documentation/architecture/09-update-architecture.md` §3: the safety copy is a *verified recent backup as a precondition*, not a new copy made for the update; the guest-snapshot idea is to be spiked before anything is built on it. The text below is kept as the record of what was asked.
 
-**If you do nothing:** both keep showing. Ranked low on purpose. *(register: R-303)*
+   *Original question:* **Where should the safety go before an app updates? This is the one decision from today's
+   measurement, and it is a design choice, not a bug report.**
+
+   **What I measured.** The box downloads new app versions by itself every 15 minutes and writes them
+   into the customer's files, whether the app is running or not. Nothing tells the customer. Then the
+   **Restart** button — not just Update — installs that new version. I watched it download a version
+   that was not on the machine and swap the app onto it in 18 seconds. **And the box does it on its
+   own** when an app fails to come back after a crash: nobody pressed anything.
+
+   **One fear is smaller than we thought, and you should have that too.** A plain power cut does
+   **not** upgrade anything. The apps come back on their old version. It only happens when an app
+   fails to return.
+
+   **One fear is bigger.** I tested whether we can undo an app update. **We cannot.** Once an app has
+   moved its data to the new version, putting the old version back gives an app that will not start at
+   all. So **"rollback" is the wrong word** and I have struck it. The only way back is to restore the
+   customer's data from a copy taken *before* the update — and today no update takes one.
+
+   **The decision, in one sentence: should the safety copy sit under the Update button only, or under
+   everything that can install a new version?**
+
+   - **Under the button only.** Cheap and quick. Covers the case a customer causes. **Leaves the
+     unattended path uncovered** — the one where an app that failed to come back is upgraded with
+     nobody watching.
+   - **Under everything.** Covers all of it. Costs more, and it has a hard limit I measured: for a
+     big app a copy is roughly 30 minutes and about twice the app's size, against a standard box that
+     ships with 20 GB. **A copy of a large app does not fit.** So this option cannot be built without
+     also answering where the copy lives.
+
+   **My pick: under everything — but decide the "where does it live" question first**, because the
+   answer decides whether the rest is even buildable.
+
+   **If you do nothing:** nothing breaks today, and no customer is at risk this week — the fleet is
+   young and its running versions match the catalog. But the exposure is real and dated: **Peti's box
+   has a one-major upgrade of `rallly` queued behind its next boot**, from a catalog change made three
+   days after it went quiet. **Who else is blocked:** nobody can spec the safe-update work until this
+   is answered, because the two options produce different products.
+
+   Full measurement, with the controls and the quoted output:
+   `felhom.eu/documentation/audits/SPIKE-app-update-2026-09-01.md`.
+
+9. **Nothing here needs you. Two things I got wrong, both now fixed.**
+
+   **You found the first one.** OpenGist showed no label. That was not a misunderstanding — the label
+   only appeared on an app that something had restarted, so an app that simply runs showed **nothing,
+   possibly for months**. On a quiet machine that is every app, which is exactly the machine we most
+   want to be able to look at. **Now the box reads what every app is on when it starts up, and writes
+   it down.** It only looks — it starts nothing and changes no app. Live on both machines: all nine
+   apps on the HP now carry a label, and so does OpenGist.
+
+   **One thing to expect, so it does not look broken:** the label says **„Naprakész"** when an app is
+   on the newest version, with **no number at all**. A number only appears when the app is *behind* —
+   and then it is how long the newer version has been waiting, not how old the running one is. That
+   was your ruling and I think it is still the right one.
+
+   **The second one was mine and smaller:** a test I wrote pinned a date and an age, so it passed the
+   day I wrote it and failed the next morning. Fixed, and I have written down that the same trap may
+   sit in six other test files — named, not accused; someone has to read them.
+
+   The box now writes down which version of each app it is really running, and shows the customer one
+   small label: **„Naprakész"** or **„Frissítés elérhető — 52 napja"**. No version numbers — a
+   household cannot act on `26.05.2`.
+
+   **Both halves are proven on the real machine.** I made two apps fail to come back, the box repaired
+   them itself and wrote down exactly what it installed — one line per container, with the fingerprint
+   that cannot lie. I checked those fingerprints against the machine independently and they match. Then
+   I opened the real customer pages and read the labels off them.
+
+   **Where I went wrong, because you should not have to spot it twice.** I told you the saved password
+   no longer worked on either machine. **It worked fine.** I read it out of the file wrongly — the
+   value is wrapped in quote marks and I only removed one kind. You caught it in one line. The machine
+   told me "wrong password", which was true, and I took it to mean the password was wrong when it meant
+   *what I sent* was wrong. I have filed that as a small job for myself: one shared way of reading that
+   file, so the next session cannot get it half right.
+
+   **If you do nothing:** nothing. This one is closed.
+
+10. **Nothing here needs you. The version of an app is now frozen, and only the Update button moves
+    it.**
+
+    **What used to happen.** The box downloaded the catalog every 15 minutes and wrote each app's new
+    definition straight over yours — including a new version. From then on, **thirteen** different
+    things could install that new version: the Update button, the Restart button, or one of eleven
+    repairs the box performs on its own. Nobody had to press anything.
+
+    **What happens now.** The version is pinned to what you have. Only a deliberate Update moves it.
+
+    **What deliberately did NOT change, because it was worth keeping.** Corrections to an app's
+    definition — a fixed health check, a memory limit, a new setting — still arrive on the same
+    15-minute cycle, and a broken definition still repairs itself. That was a real benefit of the old
+    behaviour and it is intact. In one sentence: **while the catalog offers the same version you are
+    running, its fixes reach you; the moment it moves to a newer version, you stay where you are until
+    you choose to update.**
+
+    **Proven on the real machine**, twice over: I pushed a genuine catalog change with no version in
+    it and watched it arrive on the normal cycle; then I pushed a version change and watched the app
+    refuse it.
+
+    **Two things this did NOT do, so they are not read as done.** The Update button is exactly as safe
+    as it was yesterday — no backup, no undo. That is the next piece of work, and it is item 7. And an
+    app the box could not confidently pin would have been left behaving exactly as before, loudly; on
+    the HP that was none of the nine.
+
+    **One rough edge I chose to write down rather than fix:** a frozen app still receives the small
+    metadata file that carries its health check, so it can be given a check written for a newer
+    version and look unwell when it is fine. **It cannot lose data — the worst case is a false
+    alarm.** Freezing that file too would break the „Frissítés elérhető" label, which is a worse trade.
+
+11. **How wide should I take the upgrade testing? This is the one decision from today, and it is
+    about money and time, not about safety.**
+
+    **What I built.** A machine that installs an app, puts real data in through the app's own front
+    door, upgrades it, and then asks the app for the data back. Not "did it start" — the box has
+    already fooled us that way once.
+
+    **I also taught it to fail.** Before believing anything, I pointed it at an upgrade I knew was
+    broken. It came back red. That is why I trust the greens.
+
+    **What it found, on three apps and five real upgrades.**
+    - **The data survived every single time.** That is the good news and it is worth having.
+    - **Whether an upgrade can be UNDONE depends on the app, not on upgrades.** Docmost will not go
+      back — the old version refuses to start on the changed data. PrivateBin goes back fine. **We
+      had been assuming one answer for all 53 apps. There isn't one.**
+    - **And it found a real problem in our own BookStack setup.** Our template moves the database
+      engine to a new major version, and the engine says, in its own words, that the conversion it
+      needs is being *skipped*. It works today. **I did not measure whether it ever breaks**, and I
+      am not going to guess.
+
+    **The decision: how many of the 53 apps do I test?**
+
+    - **Only the apps that keep data in a database (~25).** Today's run showed the undo question only
+      ever bites there. **Cost: roughly a day of my time.**
+    - **All 53.** Complete, and it gives us a list nobody has. **Cost: several days**, and the reason
+      is not the machines — it is that **each app needs its own hand-written way in**, and that work
+      does not get cheaper the more you do. Two of today's three needed one, and one of them took two
+      attempts.
+
+    **My pick: the database ones first.** It answers the question that changes the product, and if it
+    goes well the rest is a decision you can take later with better numbers.
+
+    **If you do nothing:** nothing breaks. The machine is built and committed, so it does not go
+    stale, and the BookStack problem is written down and waiting either way.
+
+12. **Shall I tell the database engine to finish its own conversion? One yes/no. It affects four
+    apps and I have measured both sides.**
+
+    **The background, in one line.** This morning I found that when BookStack's database engine moves
+    to a new major version, the engine says the conversion it needs is being **skipped** — and then
+    works anyway.
+
+    **What I measured this afternoon.**
+    - **It does not get worse.** I restarted it five times. The app answered correctly all five
+      times, and nothing changed for the worse.
+    - **But it never fixes itself either.** Every single time it starts, the engine says the check is
+      still needed. It will say that forever.
+    - **Fixing it takes seven seconds.** The engine backs itself up first, does the conversion, done.
+    - **And the catch I was worried about is not there.** I expected that converting properly would
+      mean we could no longer go back to the old version. **We still can** — I tried it, and the app
+      came back with its data.
+
+    **So this is not the difficult trade I thought I would be bringing you. It is a cheap tidy-up.**
+
+    - **Yes, turn it on.** Four apps get a correctly-converted database — BookStack, Kimai, Nextcloud
+      and RomM. Costs seven seconds the one time each of them moves a major version. **This is what I
+      would do.**
+    - **No, leave it.** Nothing breaks today. The engine keeps saying it wants a check, on every
+      start, indefinitely.
+
+    **One honest limit on my "yes".** I proved the app's normal use keeps working. I did **not** test
+    every database feature on an unconverted engine — so I can tell you it has not broken, not that
+    nothing can. That is exactly why leaving it alone is a real choice and not just laziness.
+
+    **If you do nothing:** nothing breaks and nothing is at risk this week. Only BookStack has
+    actually moved a major version so far; the other three will land in the same place when their
+    turn comes. I have not changed any template — this is your call, not a quiet edit.
+
+13. **"Delete my data too" now deletes the data — or tells you it could not.** Until today, when a customer removed an app and ticked the box, the box said it worked and left everything on the drive (128 MB of a Nextcloud on 2026-09-01). The cause: the removal asked one global setting for the drive, and no machine fills that setting in. Every other part of the box already asks the app itself where its data is. Now the removal does too. If the box cannot work out where the data is, it refuses and keeps the app, so you can try again — it never again reports success over data left behind. Proven on the HP with a throwaway Nextcloud: 63 MB the app wrote itself was gone after removal, and the answer listed it; the refusal was shown with the app still in place; an app with no drive data gets a plain "nothing to delete" note. Live on both machines (0.236.0). No standing app was touched. **If you do nothing:** nothing to do.
+
+8. **`demo-hp`'s network setup does not match our own notes** (R-338) — the machine works, the page is
+   wrong, or the other way round. **If you do nothing:** the page keeps misleading the next session,
+   as it misled one by an hour.
+
+## Decided — and what would reopen each
+
+- **THE BACKUP AND RESTORE WORK IS FINISHED FOR BETA. DECIDED 2026-09-01.**
+  **What is done, and proven on the real machines:** everything **you or a customer** does alone —
+  getting deleted files back, getting an app's data back, getting a whole app back, and losing a
+  drive. The restore tells you what it put back, refuses if there is no room, will not accept a
+  half-copy, and puts your own data back if it fails.
+  **What is parked until after beta:** everything **only I do, with you** — rebuilding a machine as
+  itself, losing a whole box, recovering from ransomware, restoring the hub, and losing Hetzner.
+  **Six of these have never been timed, and the hub has never been restored.** They are written down,
+  they are real, and **none of them stops a beta customer.**
+  **Reopens if:** something a customer does for themselves turns out to be broken; **or** Hetzner's
+  two answers change what the snapshots are worth; **or** a real customer's data is at stake in one
+  of the parked items.
+
+- **The alarm that promised too much: FIXED and live (hub 0.111.1). DECIDED 2026-09-01.**
+  Yesterday's alarm mail said a deleted backup was *"recoverable file-by-file"*. **We now know it is
+  not.** I removed the promise rather than writing a new one, so the sentence stays true whatever
+  Hetzner answers. It no longer says the data is lost either — that is still usually untrue.
+  **Reopens if:** Hetzner's answers give us a real route back; then the alarm can name it.
+
+
+- **The missing-golden warning is now pointed at the people who can act on it (R-404). DECIDED
+  2026-09-01, and built the same day.** The warning was aimed at the wrong repository: the one where
+  a release actually happens never checked at all, while the one that only holds documents was
+  refused on every push — including the push that RECORDS a golden bake, which is the very act that
+  clears the warning. So the check was blocking its own cure, and we had skipped it thirteen times.
+  **What changed:** the release repository now prints a reminder the moment a release is committed
+  (it never blocks — you cannot bake a golden for a version you have not pushed yet), and the
+  documents repository still runs the check on every push and still says so loudly, but only refuses
+  a push that touches real code. **Every other check still blocks everything, always.** Nothing was
+  silenced and no product code changed. **Reopens if:** a release ever ships without a golden and
+  nobody noticed — that would mean the reminder is not reaching anyone, and the answer would be to
+  make the release repository refuse rather than remind.
+
+- **Getting old backups back yourself: NOT BUILT, deliberately.** **Reopens if:** a real customer
+  asks. *(R-312)*
+- **The unopenable old copy on `demo-felhom`: KEPT as a test fixture** — the only state in existence
+  where a set-aside store is present and cannot be opened. **Delete when:** that work ships or is
+  abandoned. *(R-313)*
+- **A machine in two kinds of trouble says both things: LEFT AS IT IS.** **Reopens if:** observed
+  outside a constructed test. *(R-303)*
 
 ## What works
 
-Both demo machines are home, healthy and reporting on the approved pair — **controller 0.214.0, agent
-0.129.0**, delivered by the floor rather than by hand. Off-site is credentialed on `demo-hp` and its
-repository still opens with the machine's own key. `drill-r50` is reverted to `virgin`, powered off.
+Both demo machines are home, healthy and reporting — agent **0.130.0** published and running on both.
+**Which controller each box runs, and where the floor sits, is item 1 above and is not restated here** —
+R-395: this paragraph carried a second copy of those numbers, it went stale by seven releases, and the
+page then disagreed with itself about the thing an operator checks first. Ask the hub (`/hosts`,
+`/configs`) or the box for what is live; a doc is never the authority on a version.
+Off-site is credentialed on `demo-hp` and its store opens with the machine's own key.
+
+**The fleet, because two summaries have been misread:** five customer records, three machines.
+`demo-felhom` and `demo-hp` are ours and disposable; `drill-r50` is a nested drill VM, reverted and
+off. **`peti-felhom` is a real machine we have not heard from since 15 July** and has no host record.
+**`tester-1` is a record with no machine.**
 
 ## Shipped
 
-- **The drive can be re-attached after a reinstall** (R-280). The restore page said *"this is two
-  clicks"* over an empty list; it was zero clicks and needed an internal path no customer could produce.
-- **The orphan card stops promising** that set-aside off-site copies can be reopened — twice over
-  (R-294, then **R-299**, which was the same claim in the plural, in the *always-visible* half, missed
-  because the guard matched one inflection of a Hungarian verb).
-- **The countdown banner stops promising retrieval it cannot see is still true** (R-302). The promise
-  is now conditional on the hub still holding the package it held when the customer decided — pinned
-  then, compared now. A sweep found the same claim in five places; a fourth was fixed with it and a
-  fifth deliberately left, because it is true where it renders.
-- **One name per secret, box side** (R-295): the dashboard code is „Beállító kód" everywhere;
-  „Visszaállító kód" is retired. It collided with the escrow „Helyreállítási kód" and cost a real code.
-- **A correct recovery code is no longer called wrong** (R-311, three components). If a customer types
-  the code for an older set of backups, the machine now checks the packages we kept, recognises it, and
-  says so: *your code is correct, it belongs to an earlier package, we kept it, your current backups are
-  fine, write to us*. It deliberately promises no restore, because there is no button yet.
-- **The countdown on `demo-felhom` is cancelled** on your ruling (R-307). Nothing was deleted; the
-  24 August deadline is gone. See R-313 for what that copy turns out to be.
-- **Vouched and delivered 2026-08-12**: golden 0.214.0, agent 0.129.0, floor 0.214.0 — both machines
-  took it themselves. The version guard was watched working on the way: `demo-hp` was **held** back
-  while its agent was older, and updated 6 seconds after the agent caught up. That guard exists because
-  a machine once ran ahead of its agent and a customer was told a correct code was wrong; first sighting.
-- **Both installer fixes are now PUBLISHED** as `installer-v1.27.0` (R-297 + R-300). Each fault was
-  watched happening first, on a machine reset to factory state: the old installer really did build a
-  machine on a base image from July, and our own uninstall really did block our own next install.
+- **Something finally checks that the off-site copies are still there and readable** (R-359 + R-397,
+  controller 0.227.1, proven on `demo-hp`). Until today **nothing did** — not the box, not the agent.
+  The whole-machine backups had their own checks; the copies holding your customers' documents and
+  photos had none, so we would have found a problem at restore time, with a customer waiting.
+  Now the box checks its own off-site store about once a week and tells you only if something is
+  wrong. **A pass sends no e-mail, on purpose** — a weekly "everything is fine" is how people stop
+  reading their alerts. **It also catches itself up:** it asks „has it been more than seven days?",
+  not „is it Sunday?", so a machine that was switched off on its check day is checked the next day.
+  **And it never gets in the backup's way** — if a backup or restore is running, the check steps aside
+  and tries again tomorrow. That is not politeness: the tool would otherwise clear a lock that a live
+  backup was holding. Proven live, twice over — a real check against your real store (35 seconds), and
+  a second check fired during the first, which correctly stepped aside without doing anything.
+  **Read item 2 under „Waiting on you" for what this check does NOT see.**
+- **The product stopped claiming a check it never ran.** The monitoring page said an integrity check
+  ran every Sunday. It did not exist. The e-mail text, the settings checkbox, the hub's side of it and
+  a debug button were all built and wired to nothing (R-397). They now have the missing piece.
+
+- **Taking the safety copy no longer destroys the app's own backup** (R-361, controller 0.221.1,
+  proven on `demo-hp`). Before every restore the machine saves a copy of your live database. To do
+  that it called the ordinary backup routine — **which always writes to the app's normal backup
+  filename first** — so the app's real backup was overwritten and then renamed away. Until the next
+  nightly run the app had **no database backup of its own**, and a local recovery in that window would
+  have told you the app never had a database. A comment in the code said this could not happen; it
+  could, and had been happening for four months. Proven fixed the only way it can be: the app's own
+  backup file is now **byte-identical before and after a restore**, on both database types.
+- **A failed database restore now puts your data back by itself** (R-379/R-380, controller 0.220.2,
+  proven on `demo-hp`). Until today, if a restore of an app's database went wrong, the machine had
+  already taken a copy of your live database — a good copy — and **nothing in the product could put it
+  back.** You were shown a filename. On one of the two database types it was worse: part of the
+  restore applied, part did not, and **the dashboard said the app was healthy**. Now the machine puts
+  your own copy back automatically and says plainly: the restore failed, your data is as it was, the
+  app is running. Proven on both database types, byte-identical both times.
+  **If even that fails**, the app is deliberately **stopped and held** rather than started — a running
+  app on a half-written database lets you type into it and makes the damage permanent — and you are
+  told to contact us. That was your ruling this morning. **Two things also stopped:** the error no
+  longer pastes raw database text at you (it was 615 bytes once, including rows out of your own
+  database), and the undo copies no longer pile up forever — three per app, and they were being copied
+  off-site permanently.
+- **An empty package can no longer wipe out a good one** (R-403, controller 0.230.0). Yesterday we
+  wrote this down as *suspected* and said plainly it had not been tested. **We tested it first, and it
+  was real.** On a demo machine, on yesterday's build: an app's copy on the second drive went from
+  **120 MB — four database backups and three data archives — to 7 KB, nothing left**, in a single
+  nightly run, and the run reported success. The cause was that the nightly job only asked *does the
+  folder exist* before copying over it, and an empty package is a folder that exists.
+  Now the nightly job refuses to replace a **complete** package with an **empty** one. It keeps what it
+  has, says so on the app's own backup page, and carries on with everything else. Proven on the same
+  machine, in the same state: **all seven files still there, byte for byte.**
+  Two more things came with it. The page no longer calls that copy fresh when the run did not refresh
+  it — it names the real date of the package instead. And after a restore from the second drive, the
+  first drive's package is filled back in immediately, so the empty state that started all this cannot
+  happen again. A copy that legitimately gets smaller still gets smaller; only the one dangerous case
+  is fenced.
+- **The copy on the second drive can now bring an app back** (R-102 + R-103, controller 0.229.0,
+  proven on `demo-hp`, **and delivered** — golden 0.229.0 is vouched and the fleet floor is raised, so
+  a machine installed today has it). Every night the box copied each app's whole recovery package onto the second
+  drive — its settings, its database and its data. It did that for months. **Nothing could open those
+  copies.** No button, no screen, no command. That mattered most in the one fault the second drive
+  exists for: if the first drive dies, the package on it dies too, and the copy that survived could
+  not be read. For **45** of the 53 apps that is everything they own.
+  Now the same restore that always worked from the first drive can read the copy on the second one,
+  and the button is on the app's own backup row. **Proved with the first drive's package taken away:**
+  Docmost came back in 29 seconds — its database, all three data volumes, a file with a Hungarian
+  accented name back byte for byte, and the app then read its own rows with its own password. Done
+  again with the app's password file also taken away: the copy carried the passwords too (2 of 2).
+  The screen that used to say „press that other button on another page" now offers the action itself.
+  It says plainly that **this one overwrites** what is there — the gentle „Fájlok visszaállítása"
+  beside it still only adds back missing files — and it names the date of the copy, so nobody puts
+  last week over today by accident.
+- **We counted the apps this affects, and settled it.** Two of our own notes disagreed — 43 or 45.
+  The answer is **45**, counted with the product's own rule against the live catalogue. The older
+  count missed **radarr and sonarr**.
+- **The off-site restore now works for the other 40 apps** (R-356, controller 0.219.0, proven on
+  `demo-hp`). It used to refuse before starting, tell the customer a running app „nincs telepítve",
+  and send them to reinstall it "to the same place" — a place those 40 apps never offer, because they
+  were never given a drive to choose. It was asking one question to answer two. Proven today on
+  `privatebin`: data planted through the app itself, backed up, **deleted**, restored — **all 15 files
+  back byte for byte**, Hungarian accented names included, message „0 fájl és 1 adatkötet
+  visszaállítva". The 13 apps that do have a drive are unchanged, checked the same way.
+- **The off-site restore gives an app's data back at all** (R-354, controller 0.218.0). It used to say
+  „0 fájl visszaállítva", report success, and the folder was simply not there. It now names what came
+  back, because a restore that mentions only its file count is how a silent loss reads as a success.
+- **Paperless's database is in the backup, and restoring it takes an undo copy first** (R-355,
+  controller 0.218.0). The dump was landing in a folder named after an app that does not exist. **One
+  app of 53 was affected**, established with a check first proved able to catch a planted second case.
+- **The system tells you when it cannot see the off-site copies** (R-339) — a mail after ~30 minutes,
+  hourly while it lasts, one all-clear. **Caveat:** it watches whether the machine answers, so it
+  would *not* have caught the 18 August fault, where one service was wedged and the machine stayed
+  healthy.
+- **The connection leak was ours and is fixed** (R-344). Our agent opened a connection to the off-site
+  box every 15 minutes and never closed it; the idle timer was switched off. Proved by fixing one
+  machine and leaving the other: same work, 4 more leaked on the untouched one, none on the fixed one.
+  The off-site box is back to **17** open connections from **415**.
+- **A dated check can no longer be quietly missed** (R-341) — but it speaks on the next push, not on
+  the day. **A machine we tell to be quiet is no longer reported as dead** (R-321). **One name per
+  secret** (R-295, R-323). **The hub's own words are under a guard** (R-324). **Removal reverses the
+  installation** (R-316). **A correct recovery code is no longer called wrong** (R-311). **The drive
+  can be re-attached after a reinstall** (R-280).
 
 ## Broken, or knowingly incomplete
 
-- **The tester's machine has no recovery route at all** — see the `PETI` row. Its host record was
-  deleted on 15 July; there is no key, no off-site copy and no local backup. **If that drive fails,
-  everything on it is lost.** First act of the visit: copy the ~3.6 GB off before anything is
-  reinstalled — it is currently the only copy in existence. Whether it stays parked is your call and is
-  deliberately left open.
-- **Kept backups can be opened — but still only by us** (R-304 partly closed, R-312 open). The machine
-  now recognises an older code and says so plainly instead of hedging. What it still cannot do is hand
-  the customer their old files: that needs the restore code to accept a second location, which is real
-  work rather than wiring. Today the honest answer is "your code is right, write to us" — and we can.
-- **The dnsmasq fix helps a machine once** (R-305). On a machine that never had Felhom it works. On the
-  second reinstall the leftover comes back, because the package is never removed — so the machine looks,
-  to our own installer, as if the household had installed it. Watched happening the same afternoon.
-- **The hub half of the naming is undone** (R-295 PARTIAL): the emails still use the retired name and
-  send people to a page a rebuilt machine does not show.
-- **The storage page has its own separate reason for showing an empty list** (R-298), untouched.
+- **We do not know what the deep check costs on a BIG store** (R-401). Since 0.228.0 the weekly check
+  re-reads **all** your stored data, not just the list of it. We had to: a copy was damaged in a way
+  that left its size unchanged, and the old shallow check said „no errors were found". Only the deep
+  check caught it. **The cost we measured was four seconds** — 35.0 s before, 39.2 s after — but that
+  was on a 134 MB store, and it will not stay four seconds. So the box now tells us: any check that
+  takes longer than five minutes writes a warning naming this item. **If you do nothing:** every
+  machine re-reads its whole store every week, however large it grows, and the first person to notice
+  would be a customer whose upload is busy. The warning is there so that does not happen.
+- **We ask the off-site box a question about once a second** (R-336) — ~85,000 a day for a box we
+  write to weekly. The leak that made this dangerous is fixed (R-344); the volume is not. The ceiling
+  is **under a year** away on the corrected measurement, not two.
+- **Peti's machine has no recovery route at all.** A real machine belonging to a real person, silent
+  since 15 July, no key, no off-site copy, no local backup. **If that drive fails, everything on it is
+  lost.** First act of any visit: copy the ~3.6 GB off before anything is reinstalled.
+- **The agent picks dnsmasq by looking at a file another package owns** (R-317) — one line; LAN name
+  resolution goes missing quietly.
+- **Three facts the machines send still have no reader** (R-264); **the storage page has its own
+  reason for an empty list** (R-298); **two thirds of the standing picture is unproven** (R-326:
+  23 of 55 claims walked — `python3 scripts/unproven.py`); **the picture still describes one defect we
+  fixed twice** (R-327).
 
 ## Working on next
 
-R-312's shape (the button, or deliberately no button); then R-305, because the tester's second
-reinstall still hits the dnsmasq wall; then the hub naming; then the 2026-08-09 batch
-(R-279 … R-292), still untriaged against everything since.
+The 2026-08-09 batch (R-279 … R-292), still untriaged; the three remaining R-264 readers; R-317 (one
+line in the agent); R-327 (decide the naming claim's status); R-359 (nothing reads the off-site store).

documentation/PROMPT-TEMPLATE.md

@@ -130,8 +130,35 @@ never park it on a branch.
    conventions, access). Versioned copy: `felhom.eu/documentation/runbooks/workspace-CLAUDE.md`.
 2. `<repo>/CLAUDE.md` — repo build/deploy, code-quality rules, trunk-based + live-validation rules.
 3. `<repo>/CONTEXT.md` — current project state / decisions / roadmap.
-4. `felhom.eu/documentation/architecture/02-controller-module-map.md` — KEEP/PORT/DELETE/MODIFY
-   per-package classification. **Read before touching `backup/`, `storage/`, `system/`, `config/`.**
+4. **THE ARCHITECTURE DOCUMENT FOR THE AREA THIS TASK TOUCHES — NAME IT AND SAY WHAT IT SAYS.**
+   Not "read the architecture folder": name the file, and state in one line what it rules about this
+   area. **A prompt that cannot name one says so explicitly, and that absence is itself recorded** —
+   an undocumented architectural decision is how a deliberate design gets "fixed" by someone who did
+   not know it was one.
+
+   | area | file |
+   |---|---|
+   | what the platform does today, per scenario | `00-capability-map.md` |
+   | topology, trust, **app-data placement (hot vs bulk)**, backup scoping | `01-topology-and-trust.md` |
+   | controller packages: KEEP/PORT/DELETE/MODIFY | `02-controller-module-map.md` |
+   | the host agent | `03-host-agent.md` |
+   | control-plane authorization | `04-control-plane-authorization.md` |
+   | the hub | `05-hub-architecture.md` |
+   | off-site connectivity | `06-offsite-connectivity.md` |
+   | tiers, capture sets, restore paths, recovery model | `07-backup-architecture.md` |
+
+   **Three sources, in this order, before any claim: the architecture folder holds the REASONING, the
+   register holds the WORK, source holds the TRUTH.** Skipping the first is how a decision gets
+   reported as a bug.
+
+   **And the test that catches it: _is what I am about to call a defect something we chose?_** If it
+   was chosen and the choice is wrong, that is **a proposal to change a decision** — it goes to the
+   operator as a decision, not filed as a bug. **Cost of learning this (R-370):** between 19 and 22
+   August a documented placement decision was called a defect in four places, because the register and
+   live source were read and `documentation/architecture/` was not.
+
+   `02-controller-module-map.md` remains **required reading before touching `backup/`, `storage/`,
+   `system/`, `config/`.**
 5. `felhom.eu/documentation/controller/<feature>.md` — code-verified feature docs (authoritative; match
    code, not summaries, if they drift).
 6. `controller/README.md` (or `felhom-agent/README.md`) — module map, feature reference, REST API.
@@ -229,7 +256,8 @@ Then:  [exact refusal — HTTP status, error, and the proven non-effect, e.g. "m
    `go build ./... && go vet ./... && go test ./...` — all green before proceeding. The build IS the
    typecheck; do not accumulate compile errors.
 2. **Minimal changes:** build only what's listed. No "while I'm here" refactors. Note anything worth
-   fixing under "Observations" (§15) — don't act on it.
+   fixing under "Observations" (§15) — don't act on it, but **do file it**: §15.9's marker rule means
+   "not acted on" never means "not recorded".
 3. **No silent failures:** never swallow a parse/exec error — log it. Check a subprocess's **own** exit
    code; never pipe in a way that hides a 127. (The silent `.felhom.yml` quoting bug + the spike's
    exit-swallow lesson.)
@@ -347,11 +375,50 @@ or **what is open changed**), update **all four** in the SAME session:
   that changes what is open without touching it re-creates exactly the thread-loss the register was
   built to solve: `REPORT.md` is overwritten every session, so nothing durable may live only there.
 
+  > **AN ENUMERATED GAP BECOMES A ROW, IN THE SAME SESSION. PROSE IS NOT A RECORD.**
+  >
+  > This binds **surveys, inventories, spikes, reviews and diagnoses**, not only implementation
+  > sessions — those are the documents that enumerate gaps, and they are the ones that have lost them.
+  > If a document says a thing is missing, unhandled, unreachable or *"not currently filed"*, it does
+  > not leave the session as prose. It leaves as a row here, with a rank and an owner. Writing
+  > *"not filed"* is not a disposition; it is a note that the work was seen and dropped.
+  >
+  > **A row in `ROADMAP.md` alone does not satisfy this.** Both files hold open work and only this one
+  > calls itself the source of truth, so a finding recorded solely there is invisible to every
+  > standing rule that says *"grep the register before minting"* (**R-369**).
+  >
+  > **The cost, recorded so the rule can be narrowed later rather than becoming permanent by
+  > accident:** R-107 — *"no offsite action unpacks the named-volume tars Tier-3 captures"* — was
+  > enumerated on **2026-07-28**, given a number, written into `ROADMAP.md` and
+  > `07-backup-architecture.md`, and never entered here. **It was rediscovered from scratch 25 days
+  > later by an overnight drill that planted files and watched them not come back**, and shipped as
+  > R-354. The work was right the first time; only the filing was missing.
+
 **Report which `OPEN-ITEMS.md` rows the task opened, closed or re-ranked** (§15). Every row carries
 an owner — a row nobody owns is how items got lost in the first place.
 
 ### N.6 Website version bump (if controller/hub version is shown on the site).
 
+### N.7 Housekeeping — before the report, not after (2026-08-22 ruling)
+
+**Nothing was ever pruned and the numbers got bad:** the register reached 672 KB across 286 entries,
+over half of it finished work, one entry at 16 KB. **A file that cannot be read is a file that cannot
+be checked**, and this project has paid for that twice — a record nobody could find because it sat
+inside an entry about something else, and a finding rediscovered because nobody could see it.
+
+1. **Compress what this session closed.** A closed row keeps its title, the version it shipped in, its
+   evidence paths, and any sentence stating a rule. Everything else goes, and it moves to
+   `backlog/CLOSED-ITEMS.md`. **Nothing is deleted:** the compressed entry names the commit whose
+   `git show` returns the full original text. **Open rows are not touched — their detail is doing a job.**
+2. **Rehome live reasoning before compressing it away.** If a closed entry carries the reason a rule
+   exists or a fence sits where it does, that reasoning moves — to `CONTEXT.md` if it is a decision,
+   to the owning `architecture/*.md` if it is a shape. **Where it is a decision, mark the resulting
+   shape `[DESIGN]` in the architecture document and point it at the log entry** (§4's map).
+   **Losing a reason is how a deliberate design becomes a bug in someone's eyes** — that cost four
+   mis-filed defect reports in August 2026 (R-370, R-376).
+3. **State the register's size in the report, before and after.** A number every session is what makes
+   growth visible; prose about tidiness is not a mechanism.
+
 ---
 
 ## 11. Tests
@@ -476,11 +543,33 @@ Report MUST include:
 6. **Deployed versions** + `docker ps` / agent-service / hub-pod verification output.
 7. **NOT yet live-validated — awaiting supervised [Bx]:** explicit list (the real
    put-data → operate → integrity flow).
-8. **Teardown evidence — all three §13 layers**, if the run provisioned anything: the machine deleted,
+8. **Evidence copied off BEFORE each revert** — for every phase that ran on a machine, the logs were
+   pulled to the evidence directory **at the end of that phase**, before any revert, snapshot restore
+   or teardown, **including the intermediate ones**. *The intermediate revert is the one that gets
+   forgotten: two sessions lost a phase's logs to a mid-run revert to `virgin` on 2026-08-12 and
+   2026-08-13 — same machine, same point, three days apart (R-320).* **If a phase's evidence is
+   already gone, the report says so plainly and the finding is REPRODUCED independently** — that is the
+   expectation, not an improvisation. A quotation read live and no longer re-readable is named as such.
+9. **Teardown evidence — all three §13 layers**, if the run provisioned anything: the machine deleted,
    `pvesm status` before/after with the space returned, and the **hub-side record's disposition named**
    (deleted / retained-with-reason / gate-blocked-with-the-command). A run that provisioned nothing says
    so. "Teardown clean" without layer 3 is not a report — it is the `sess-c` failure.
-9. **Observations:** out-of-scope items noticed — documented, NOT acted on.
+9. **Observations:** out-of-scope items noticed. **Every item carries `FILED: R-NNN` naming the
+   register row opened for it in THIS session, or `NOT-A-FINDING: <reason>` declaring plainly that it
+   does not warrant one.** Opening the row is the default; declaring is the exception and its reason
+   is the whole of the marker.
+
+   **This wording replaces "documented, NOT acted on" (2026-08-24, R-389), and the old wording was
+   the defect.** "Documented" was satisfied by a paragraph — and `REPORT.md` is overwritten every
+   session, so a paragraph has a lifetime of one session. On 2026-08-23 a live, reproducible finding
+   (only the first broken app per hour reaches the operator) was written under Observations and
+   nowhere else; it had no register row and had to be re-derived the next day. That is the same shape
+   as R-341, and this project's own standard says **a rule without a mechanism is a wish**.
+
+   **The mechanism is gate 11** (`scripts/observations_gate.py`, registered in the repo runners),
+   which refuses a push whose `REPORT.md` carries an observation with neither marker. Note what it
+   deliberately does NOT accept: a passing mention of some other `R-NNN`. The lost item cited `R-182`
+   as an analogy, so "cites a register row" would have passed the very item the gate exists to catch.
 
 ---
 

documentation/architecture/00-capability-map.md

@@ -1,5 +1,20 @@
 # 00 — Felhom Capability Map
 
+> **How to read this document.** Two kinds of statement appear, and where this document marks them it
+> marks them like this — the same wording as `07-backup-architecture.md:11-17`, carried here on
+> 2026-08-22 (R-376) so a reader meets one convention and not eight:
+>
+> - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet.
+> - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation.
+>
+> **Statements in this document are NOT yet all marked.** Marking them wholesale is a large judgement
+> exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376).
+> **An unmarked statement therefore means "not yet classified", never "observed".** That ambiguity is
+> exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat
+> unmarked beside a marked `[FACT]`, and was read as an observation and reported as a defect.
+
+
+
 > **What this is:** the single cross-component truth table of what the Felhom platform can do
 > **today**, at what confidence level, with verifiable evidence. Rows are *scenarios* (user- or
 > operator-visible outcomes), not modules — a scenario spans agent + controller + hub + catalog,
@@ -40,7 +55,38 @@
 
 | **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. **R-199 BACK-POINTER (omitted when this row was written): the recovery chain's link inventory and the per-link status live in `audits/RECON-offsite-dr-chain-2026-08-04.md` §3; links 1–8 are walked, 9–11 are not.** **Re-confirmed 2026-08-04 evening by an attempt to prove the DATA half:** the R-201 drill was prepared on demo-hp and **halted before the wipe** — the sentinel file was not in the off-site snapshot (R-203), so the wipe would have destroyed it and proven nothing. **No file has still ever been restored from an off-site backup after a wipe** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). The install half of the chain (controller v0.196.0 `--recover-offsite-install`, R-200) is likewise unit-proven only — it has never run against a live recovery. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant  **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent Line truncated
 
-| **A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end** | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE (2026-08-04 night drill)** | demo-hp's controller data volume was destroyed and the sentinel deleted from disk. The customer's recovery code then produced `8a9e33aa4da6…` — **byte-identical** to the pre-wipe on-disk key and to the hub's independent record — and installed cleanly on the bare box. `identity_blob` was unchanged across the wipe (572 B, `updated_at` still 11:11:37). Evidence: `audits/DRILL-r201-night-run-2026-08-04.md` §2 | **WHAT IT DOES AND DOES NOT CLAIM.** PROVEN: after a real rebuild the key recovers byte-identical, the EXISTING repository opens (3 snapshots, 42 026 B — the pre-wipe size; not a fresh history), and the customer restore flow returns the file byte-identical. **ALL FOUR MANUAL INTERVENTIONS ARE CLOSED, AND THE CUSTOMER IS NOW OFFERED THE RECOVERY (2026-08-05). ONE QUALIFIER REMAINS AND IT IS NOT THE OLD ONE — state it precisely.** The 2026-08-04 drill needed four interventions between "the key is recoverable" and "the file is back", none of them in any design document (R-204). **Items 1–3 closed in controller v0.198.0 + hub v0.95.0** (the reset code works first time; a Re-issue no longer marks a healthy escrow stale; a unit restore states it returned the app's definition and database and NOT the customer's files). **Item 4 closed in controller v0.199.0 + hub v0.96.0** (a rebuilt box DECLARES `offsite.state=needs_credential` and `internal/offsiteheal` re-arms the stored credential before minting). **AND THE "needs someone who knows to look" HALF IS NOW GONE TOO (controller v0.200.0, R-193):** a full-page recovery screen takes over the landing pages while the hub holds a sealed package this box cannot open, explains that nobody can replace a lost recovery code, takes the code, opens the repository and lists what is in it — proven live on demo-felhom, which is genuinely in that state (`/launcher` → 302 `/recovery`; a wrong code refused with nothing written, three times, no lockout; the code found in no file, no container log and no debug ring, with a planted-copy positive control that first exposed a mis-aimed sweep). **WHAT THE QUALIFIER IS NOW, and it is narrower:** (1) **the final unlock has never been driven with a CORRECT code through the page** — no recovery code was kept for demo-felhom's orphaned history and demo-hp's is operator-held out of band, so the live run exercised the whole chain (handler → agent → hub fetch → age KDF) and stopped at the unseal; the install and listing rest on tests. (2) **putting files back in place is deliberately NOT part of this** — restore stays per-app, and the step after the listing is R-213. (3) **the journey has still not been re-walked end to end since these fixes** — the closures are proven individually, not as one uninterrupted run. That re-walk is one more drill and it is what is owed. Scope: `demo-hp`, a controller-data-volume rebuild — **NOT** a total host loss, and **NOT** a guest reprovision. **⚠ THE RE-WALK RAN ON 2026-08-05 (CAMPAIGN-11) AND IT FAILED. THIS ROW STAYS AS IT IS — it records a key recovering and a file returning, which remains true — but the JOURNEY row below is the honest verdict and this row must not be read as covering it.** The campaign built a throwaway appliance from the published ISO, destroyed it, and walked the customer's route: **the data half PASSED** (three sentinels byte-identical, including a 12 MB binary and a non-ASCII Hungarian filename, out of the pre-wipe snapshot in 16 s) and **the journey half FAILED** — four operator interventions, three needing root on the appliance, and the first of them was the machine telling the customer their perfectly correct recovery code was wrong (R-216). **R-198's retention is NO LONGER unit-proven — it was PROVEN IN PRODUCTION on 2026-08-05** (CAMPAIGN-11 Phase 3): the first supersession since the fix retained `identity_blob` at **572 B, byte-length exact**, carrying the OLD key `626e424670248db3` while the current row moved to the newly minted `e11a6c542b73477a`; `offsite_repo_key_changed` and `escrow_superseded` both fired at the instant of supersession, and the next off-site run REFUSED as `orphaned` rather than starting a fresh history. Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` |
+| **A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end** | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE (2026-08-04 night drill)** | demo-hp's controller data volume was destroyed and the sentinel deleted from disk. The customer's recovery code then produced `8a9e33aa4da6…` — **byte-identical** to the pre-wipe on-disk key and to the hub's independent record — and installed cleanly on the bare box. `identity_blob` was unchanged across the wipe (572 B, `updated_at` still 11:11:37). Evidence: `audits/DRILL-r201-night-run-2026-08-04.md` §2 | **WHAT IT DOES AND DOES NOT CLAIM.** PROVEN: after a real rebuild the key recovers byte-identical, the EXISTING repository opens (3 snapshots, 42 026 B — the pre-wipe size; not a fresh history), and the customer restore flow returns the file byte-identical. **ALL FOUR MANUAL INTERVENTIONS ARE CLOSED, AND THE CUSTOMER IS NOW OFFERED THE RECOVERY (2026-08-05). ONE QUALIFIER REMAINS AND IT IS NOT THE OLD ONE — state it precisely.** The 2026-08-04 drill needed four interventions between "the key is recoverable" and "the file is back", none of them in any design document (R-204). **Items 1–3 closed in controller v0.198.0 + hub v0.95.0** (the reset code works first time; a Re-issue no longer marks a healthy escrow stale; a unit restore states it returned the app's definition and database and NOT the customer's files). **Item 4 closed in controller v0.199.0 + hub v0.96.0** (a rebuilt box DECLARES `offsite.state=needs_credential` and `internal/offsiteheal` re-arms the stored credential before minting). **AND THE "needs someone who knows to look" HALF IS NOW GONE TOO (controller v0.200.0, R-193):** a full-page recovery screen takes over the landing pages while the hub holds a sealed package this box cannot open, explains that nobody can replace a lost recovery code, takes the code, opens the repository and lists what is in it — proven live on demo-felhom, which is genuinely in that state (`/launcher` → 302 `/recovery`; a wrong code refused with nothing written, three times, no lockout; the code found in no file, no container log and no debug ring, with a planted-copy positive control that first exposed a mis-aimed sweep). **WHAT THE QUALIFIER IS NOW, and it is narrower:** (1) **the final unlock has never been driven with a CORRECT code through the page** — no recovery code was kept for demo-felhom's orphaned history and demo-hp's is operator-held out of band, so the live run exercised the whole chain (handler → agent → hub fetch → age KDF) and stopped at the unseal; the install and listing rest on tests. (2) **putting files back in place is deliberately NOT part of this** — restore stays per-app, and the step after the listing is R-213. (3) **the journey has still not been re-walked end to end since these fixes** — the closures are proven individually, not as one uninterrupted run. That re-walk is one more drill and it is what is owed. Scope: `demo-hp`, a controller-data-volume rebuild — **NOT** a total host loss, and **NOT** a guest reprovision. **⚠ THE RE-WALK RAN ON 2026-08-05 (CAMPAIGN-11) AND IT FAILED. THIS ROW STAYS AS IT IS — it records a key recovering and a file returning, which remains true — but the JOURNEY row below is the honest verdict and this row must not be read as covering it.** The campaign built a throwaway appliance from the published ISO, destroyed it, and walked the customer's route: **the data half PASSED** (three sentinels byte-identical, including a 12 MB binary and a non-ASCII Hungarian filename, out of the pre-wipe snapshot in 16 s) and **the journey half FAILED** — four operator interventions, three needing root on the appliance, and the first of them was the machine telling the customer their perfectly correct recovery code was wrong (R-216). **R-198's retention is NO LONGER unit-proven — it was PROVEN IN PRODUCTION on 2026-08-05** (CAMPAIGN-11 Phase 3): the first supersession since the fix retained `identity_blob` at **572 B, byte-length exact**, carrying the OLD key `626e424670248db3` while the current row moved to the newly minted `e11a6c542b73477a`; `offsite_repo_key_changed` and `escrow_superseded` both fired at the instant of supersession, and the next off-site run REFUSED as `orphaned` rather than starting a fresh history. Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` **RE-WIDENED 2026-08-22, and the narrowing below is now HISTORY — read both, in order.** Both defects named in the 2026-08-21 narrowing are closed and proven live: **R-354** (controller v0.218.0, `volReplay`) and **R-356** (controller v0.219.0). The **40-class end-to-end story is now WALKED**, including the hardest ten of it: `audits/DRILL-r356-hot-only-restore-2026-08-22/` proved a driveless app with NO database (`privatebin`: planted, off-sited, deleted, restored, **15/15 files byte-identical**, two Hungarian accented names), and `audits/DRILL-r356b-drivLine truncated
+It was read from source that a HELD app would raise the dead-app banner and a customer e-mail, because
+it keeps one container (its database) and so is not `StateStopped`. **It does not.** A held app
+aggregates to `unhealthy`, and `aggregateState` checks `unhealthy > 0` before the mixed-case degraded
+branch while `IsDownState` excludes `unhealthy` entirely. Measured on `demo-hp` on the shipped
+v0.220.2: hold created 21:11:19Z, dead-app scans every 30 s ran over it, and the heartbeat reported
+`0 currently down` throughout. **No suppression was built, because there was nothing to suppress.**
+The detector itself is sound — `classifyRunStates` is pure and a `degraded`/`exited` stack does raise
+the banner, pinned by `TestClassifyRunStates_PositiveControl_ADownStackDoesAlarm`. What the same
+measurement DID expose is **R-384**: an app whose database has died is `unhealthy` too, and is
+likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decision.txt`.
+
+> **R-384 CLOSED in controller v0.222.0 (2026-08-23), proven live.** The defect was the ORDER of two
+> questions, not the `unhealthy` exclusion: `aggregateState` now asks *"is a supervised member dead?"*
+> **before** the `unhealthy`/`starting`/`restarting` returns, and "some members are up" counts any
+> member not in the down bucket rather than `running` alone. `IsDownState` is byte-identical.
+> **Measured on `demo-hp` 2026-08-23** with the same fixture that read `0 currently down` the day
+> before: `bookstack-db` stopped 05:30:07Z → `app_start_failed` fired at **05:30:14Z**, the banner read
+> *„Telepített alkalmazás nem fut: BookStack (degraded)"*, the stack read `state=degraded` **while its
+> front end was `unhealthy`**, and the heartbeat printed **`1 currently down`** against the previous
+> day's `0`. Evidence: `audits/DRILL-r384-dead-db-alarm-2026-08-23/`.
+>
+> **The HELD-app half of the paragraph above is now also covered** — a held app keeps its database
+> container, so it is the same shape and reaches the same `degraded` verdict.
+>
+> **The alarm ladder that decides all of this now has an owning document:** see
+> `08-alarm-ladder.md` (written 2026-08-23 — before that date no document owned it, and that absence
+> is why the ordering defect was legible only from source).
+
+**WHAT IS STILL NOT CLAIMED:** the FAILURE path is where this class is weak, not the success path — a corrupt dump leaves the customer with an emptied or partially-applied database and an undo copy **no product action can apply** (**R-379**), and on MariaDB it does so behind an app that reports `health=healthy` (**R-380**). The success story is proven; the recovery-from-a-bad-restore story is not.
+
+**NARROWED 2026-08-21 by the backup-truth drill — kept as history; both halves have since closed, see the entry immediately above.** Proven that night on `demo-hp` with planted, hash-recorded files: the **declared-userdata** leg of a **drive-declaring** app does come back byte-identical (`calibre-web`, 5/5 including two Hungarian accented filenames). **Two legs of the same story do NOT:** (a) the off-site restore has **no named-volume leg at all**, so an app's volume tar sits in the unit, in the snapshot and in the checking folder and is never replayed (**R-354**); (b) for the **40 of 53** apps that declare no data drive the off-site restore **refuses outright**, saying a running app „nincs telepítve" (**R-356**). Since the 40-class keeps ALL its data in named volumes, the end-to-end story is **unproven for that class and disproven for the volume leg generally**. The escrow/key half of this row is untouched by that and still stands. Evidence: `audits/DRILL-backup-truth-2026-08-21/evidence/` and `REPORT.md` (2026-08-21). |
 | **The customer's own UNAIDED recovery journey, end to end** | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE (2026-08-07, the fifth walk) — an unaided customer journey on a real installation, SCOPED: see what it does not claim** | A throwaway appliance was built from the published ISO on demo-hp, claimed, given three sentinels, escrowed, backed up off-site, then destroyed — guest purged and both enrolled drives wiped. The rebuild and the customer's route were then walked with **no command line inside the guest**. **Data: PASS.** All three sentinels byte-identical (`beb9175d…`, `7c8cb0ad…`, `e012e76f…`), including a 12 MB binary and `C11-őrszem-ékezetes-árvíztűrő.txt` — the Hungarian filename survived disk → restic → SFTP → Storage Box → restore → disk intact. Restored in **16 s** out of the pre-wipe snapshot `f3d9cd67`, non-destructively. **Journey: FAIL.** Four interventions, three of them root on the appliance: R-216 (a correct code called wrong), R-218 (the store never opened; the box stopped asking), R-219 (the promised listing structurally could not render), R-220 (the app could not be redeployed — its drives were unenrollable). Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` | **WHAT IT DOES NOT CLAIM, and what the v0.201.0/v0.97.1 fixes do and do not change.** Six of the nine findings are fixed and each is red-proofed (R-216, R-218, R-219, R-217, R-222, R-215) — but **fixes are not a journey.** This row goes green only when the walk is repeated end to end and completes with no operator intervention. **Three findings are deliberately still open and each is a live blocker for some flow:** **R-214** (the console never stops showing a stale pairing code), **R-220** (after a rebuild the customer's drives cannot be re-enrolled — the deploy refuses and the wizard's list is empty, so no app can be redeployed onto its own data drive), **R-221** (a rebuilt box cannot run the escrow ceremony at all, because the pbsdr convergence marker survives on the host while the seeded `escrow.pbs_storage_id` is rewritten out of `agent.json`). **And R-216's fix does not by itself make a NEW box work:** with the hub guard corrected but the Day-0 manifest still vouching agent 0.120.0, a new box is **HELD, not served** — it stops being lied to, which is better, but the feature does not work for it until agent 0.125.0 is vouched.  **⚠ CAMPAIGN-11 PHASES 2 + 4 RAN OVERNIGHT 2026-08-05/06 AND THIS ROW STAYS **FAIL**.** **What they add:** eleven injected faults against the recovery journey, each with a positive control, and a full unattended scheduled cycle. **The backup promise strengthened** — the off-site tier ran itself at 04:15 on a box rebuilt twice and set aside hours earlier (`snapshot_count` 1 → 2), all five daily jobs fired exactly once, and nothing on the must-not list fired. **R-217's and R-215's fixes were proven live under exactly their faults**, the set-aside proved it does not delete (12 535 KB byte-exact at the far end), and a wrong code was refused three times with nothing written and no lockout. **What they do NOT add: any progress on the JOURNEY.** These faults are **not a re-walk** — no customer route was walked end to end — and four new findings say the journey got no better where it matters: **R-224** (a hub outage and a stopped agent are BOTH reported to the customer as a bad recovery code, in 0.056 s and 0.030 s — no unseal attempted; the agent's own `err` distinguishes them and it is discarded at the HTTP boundary, and R-216's gate answers `source=version` so it cannot see reachability — **Phase 1's headline defect relocated from the version channel to the transport**), **R-226** (M1, the only message telling a customer to check their typing, is unreachable on any box that has re-escrowed), **R-225** (an unread store renders `0 pillanatkép · 0 GB` above a card saying it holds backups — 1 snapshot and 12 535 KB were really there), **R-228** (the set-aside history is recorded in `orphaned_renamed_to` and shown by nobody — `OrphanedRenamedTo` has zero references in any template or handler). **§4.1 is now MEASURED rather than deduced** — the floor IS served, from the box's own rendered `GetFloor()` and a cold-started controller's settle-gate line. **§4.2's positive half is still NOT measured** and needs a rebuild. Evidence: `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md`, `tests/campaign11-evidence-2026-08-05/journal-phase24.md`  **2026-08-06 — THE FIVE PHASE-2 FINDINGS ARE FIXED AND THIS ROW STILL SAYS FAIL.** controller **v0.202.0** + agent **v0.126.0** close R-224, R-226, R-225, R-227, R-228. **What changed:** the unlock path now classifies WHY it failed, from the value and never the text, and the message that mentions typing is reachable from exactly one class — a real refusal — with **unknown defaulting to neutral**. Proven live on the venue with the same wrong code and only the hub's reachability changLine truncated
 | **Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts** | controller v0.197.0 | **PROVEN-LIVE (2026-08-04) — and this row was OPTIMISTIC before it** | On demo-hp, an app on the **system-data fallback** bound `/mnt/sys_drive/userdata/media/books` while the capture set looked in `/mnt/sys_drive/felhom-data/userdata/media/books`: the declared-mandatory directory was in **no** snapshot and the run reported `ok` (R-203). After v0.197.0 the capture log reads `1 mandatory path(s)` and the file is **listed inside the snapshot** — `restic ls -l latest --tag calibre-web` → `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt`. Evidence: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 and the v0.197.0 CHANGELOG | **WHAT IT DOES NOT CLAIM.** It covers CAPTURE, not RESTORE: **no file has ever been restored from an off-site backup after a wipe** (R-201, ready to resume). It also does not claim the enrolled-drive layout was ever wrong — it was not, which is precisely why this survived. And a run that still misses a mandatory directory now reports **`incomplete`** rather than `ok`, so this row's guarantee is one the status can express. **Widened 2026-08-06 (controller v0.205.0, R-234):** the same verdict now also covers an app skipped ENTIRELY — until then a missing declared FOLDER made the run incomplete while an app with no recovery unit at all still reported `ok`, so the smaller gap moved the verdict and the bigger one did not. **This row still claims CAPTURE, not that a newly-selected app is protected by the next run** — for a DEPLOYED app it is (the run's own pre-dump phase writes the unit, measured 2026-08-06), and for an undeployed one it is not and the card now says so || **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** |
 | **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **THE HOST TIER IS NOW PROVEN-LIVE, UNATTENDED, ON BOTH DEMO BOXES (2026-08-0Line truncated
@@ -53,10 +99,15 @@
 | Scenario | Components | Status | Evidence | Gap / roadmap |
 |---|---|---|---|---|
 | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | `CAMPAIGN-2` T-DEPLOY-SET (7 apps, env config, health-aware); `RERUN-p1p3` (×4 PASS) | Memory-guard FIRING is not live-shown (T-RES-MEMGUARD never fired: ample RAM / auth-walled) — implemented + unit-level only |
-| App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | `CAMPAIGN-2` T-LIFECYCLE (stop/start/restart/update/logs); remove live in `CAMPAIGN-3` | Redeploy-after-remove edge remains open (T-REMOVE-REDEPLOY never cleanly passed — stale dryrun journal); non-pilot-critical |
+| App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE — the ACTIONS work. NARROWED 2026-09-13: `CAMPAIGN-3` proved `remove` removes the APP, not the DATA — the "delete my data" half was INERT on every box until controller v0.236.0 (R-442). RE-PROVEN 2026-09-13 on demo-hp: data written by the app itself (63 MB) gone after removal and listed; an unresolvable data location is REFUSED (409) with the app kept; an SSD app gets `[]` and a note.** | `CAMPAIGN-2` T-LIFECYCLE (stop/start/restart/update/logs); remove (app only) live in `CAMPAIGN-3`; **remove WITH data: `audits/R442-2026-09-13/`**; **data behaviour: `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01)** | Redeploy-after-remove edge remains open (T-REMOVE-REDEPLOY never cleanly passed — stale dryrun journal); non-pilot-critical |
+| **What `restart` and `update` do to a deployed app whose compose file the catalog already moved** | controller **v0.235.0** | **CHANGED 2026-09-06 — they NO LONGER upgrade it.** The row below records what shipped; this text records what it replaced, because every box under v0.235.0 still behaves the old way. **Up to v0.234.0: PROVEN-LIVE (2026-09-01) — they UPGRADE it.** Every lifecycle action ends in `docker compose up -d`, which makes the container match the file and PULLS the image itself when it is missing (measured: 18.3 s with a pull, 0.5 s without; negative control with an unchanged file did not even recreate the container). This is DELIBERATE on the restart path — `Manager.RestartStack` says so in a comment — but the syncer moves the file under a deployed app on a 15-minute cycle with no deployed check (R-438), and NOTHING tells the customer. | `audits/SPIKE-app-update-2026-09-01.md` §2, §3 | **No safety copy is taken by any of them** — `writeSafetyDump` is DATABASE-ONLY and is not on the update path at all. R-438, R-440, R-443. |
+| **Whether the box UPGRADES an app by itself, with nobody pressing anything** | controller | **PROVEN-LIVE (2026-09-01) — YES, but only when an app fails to come back.** A plain power cut does NOT upgrade: Docker's `restart: unless-stopped` restores the old containers and the reconciler logs `no boot-orphaned apps (nothing to start)`. When an app does NOT return, `Reconciler.Run` (`bootrecon.go:269`) calls `StartStack` -> `compose up -d` and the app comes back on the NEW version, unattended (measured). **13 non-API call sites across 9 files reach `up -d` this way** — not the five previously believed. | `audits/SPIKE-app-update-2026-09-01.md` §2, §8 | The drive-return gate (`intermediary.go:222`) and `AppStopGuard.Recover` (`appstop_marker.go:283`) call the same function; located by reading, **not exercised live** — stated as such. |
+| **Whether an app UPGRADE can be undone** | controller + catalog | **PROVEN-LIVE (2026-09-01) — NO, and "rollback" is the wrong word for it.** Once a migration has RUN, putting the old image tag back yields a container that refuses to start: Nextcloud — *"the version of the data (32.0.9.2) is higher than the docker image version (31.0.14.1) and downgrading is not supported"*. A 3-major jump is refused outright (*"only possible to upgrade one major version at a time"*) and IS recoverable, precisely because nothing migrated. Positive control: the data is not destroyed — returning to 32.0.9 restored both seeded markers byte-identical. | `audits/SPIKE-app-update-2026-09-01.md` §7 | The only route back is restoring DATA from a copy taken BEFORE the update — which no update path takes. And a restore's image-level rollback is itself overwritten by the syncer within 15 minutes (R-441). R-40 is confirmed live by the same measurement. |
 | Protected infra stacks can't be stopped/removed from UI | controller | **PROVEN-LIVE** | `CAMPAIGN-nomercy` + `RERUN-p1p3` T-SEC-PROTECTED (refuse stop/remove, stay Up) | (Cited `CAMPAIGN-2` T-SEC-PROTECTED was a stale-dryrun FAIL — corrected to the runs with a real server-side refusal) |
+| **What VERSION a box is running, and whether it is behind the catalog** | controller **v0.234.0** + catalog `69761cf` | **PROVEN-LIVE — both the record and the rendered badge.** | **`tests/VALIDATION-update-slice12-2026-09-02.md`** — on demo-hp 0.233.0, through a REAL production caller (`bootrecon → StartStack → compose up -d → recordInstalledImages`, no hand-set state): `bentopdf` recorded **1** service and `bookstack` recorded **2**, keyed by compose SERVICE name, and **all three digests match the ground truth read independently from the containers before anything was touched**. `catalog_since` reached the box on the normal 15-minute sync. bookstack's two encrypted secrets are byte-identical across the write. Badge evidence: same file §4. **v0.234.0 startup backfill, PROVEN LIVE 2026-09-03 on demo-hp:** the record was stripped from `privatebin` (1 service) and `romm` (3 services) to recreate the pre-0.233.0 shape, the controller restarted, and the backfill re-seeded **exactly** those two — every digest matching the ground truth read from the containers beforehand — while logging `2 app(s) recorded, 7 already had a record, 0 left unrecorded`. **All 9 deployed apps then carried „Naprakész" on `/stacks`** (ASCII fragments with a negative control at 0). On demo-felhom the operator's own case, OpenGist, now renders the badge. **Why the backfill exists at all: without it the label never reached an app that simply runs**, which the operator found the morning after v0.233.0. Unit side: `installed_test.go` + `updatebadge_test.go`, incl. a wiring test through a real `RestartStack`, an AST walk of all four call sites, and three companion red-proofs. | **THE UNEXERCISED LEG, NAMED: one badge STATE of four.** „Frissítés elérhető" WITHOUT an age needs an app whose `catalog_since` is absent, malformed or future-dated, and all 53 now carry a valid one — unit-tested only (`TestGroupF`). The other three are live: „Naprakész" ×2 on `/stacks` and on `/apps/bookstack`; **NO badge at all on `/apps/docmost`**, a deployed app with no record — absent is UNKNOWN and is not rendered as current; and „Frissítés elérhető — **52 napja**" on both surfaces, the age being real arithmetic on bentopdf's `catalog_since` 2026-07-12. **The behind state was staged by editing bentopdf's compose tag ONLY** — no container restarted, no `up -d` — then reverted byte-identically (`sha256` equal, `diff` empty); so the RENDER is measured and the syncer's own half stays measured separately in the spike. Searched with `grep -oF` ASCII fragments plus positive AND negative controls. **The Frissítés/Újraindítás/Leállítás buttons are unchanged on the live page in the behind state.** **Absent means UNKNOWN, never current** — a legacy `app.yaml` renders NOTHING, red-proved. **No version number is shown to the customer** and **no registry is queried**, so „Naprakész" CAN BE FALSE for the 23 floating pins (**R-446**). Nothing about updating changed: R-438, R-440, R-441, R-443 all stand. Reasoning: `architecture/09-update-architecture.md`; remaining slices R-447..R-452 |
+| **An app's VERSION is frozen to what the customer has; only a deliberate Update moves it, while template CORRECTIONS still arrive** | controller v0.235.0 | **PROVEN-LIVE (2026-09-06)** — by two REAL catalog pushes travelling the REAL 15-minute cycle, not a hand-edited file | **`tests/VALIDATION-update-slice3-2026-09-06.md`** — a non-image catalog change REACHED the pinned app (08:01:51Z) with the container untouched; an image change did NOT (08:20:29Z); and the restart afterwards took **0.1 s, did not recreate the container, and never pulled the new image**, against the spike's **18.3 s with a pull** for the identical sequence before. The Update button still moved the version (pin advanced 17 s BEFORE the pull completed) and the teardown update returned the container to the baseline digest byte for byte. The freeze holds in BOTH directions. Unit side: `internal/sync/render_test.go` (the whole render table, incl. self-healing in BOTH branches), `internal/stacks/pin_test.go` (adoption never guesses; the update advances the pin BEFORE the pull), `TestGroupG` (the badge reads the catalog, not the frozen file), `TestGroupH` (an AST walk of `cmd/controller/main.go` asserting the seam, adoption, and their ORDER against `syncer.Start()`). **Three companion red-proofs**, each run, failing, and reverted. | Operator ruling 2026-09-06, Option 1 (`architecture/09-update-architecture.md` §3.4). **Nothing was added to the thirteen `compose up -d` call sites** — most are repairs, and a repair that refuses to repair leaves an app down; they were made safe by removing the reason. `pinned_images` is INTENT, `installed_images` is an OBSERVATION — never fed from each other (R-166, one field over). **Known limitations, all recorded rather than fixed:** a frozen app is frozen WHOLE (§8.4); `.felhom.yml` keeps flowing, so a frozen app can get a probe for a newer version — false alarm, never data loss (**R-458**); and **the Update button is still unguarded** (R-448 is slice 4). Closes R-447, R-441, R-438 |
 | Catalog sync (git, 15 min) + orphan lifecycle + validation choke point (bad `backup:` block degrades to legacy, loudly) | controller v0.132, catalog | **PROVEN-LIVE** | `CAMPAIGN-2` T-SYNC-IDEMPOTENT; v0.132 LoadMetadata red-proofs | |
-| Lemez-egészség felügyelet: per-disk SMART kártya („Lemezek állapota") + degradáció-riasztás (Rendben/Figyelmeztetés/Hiba/Nincs adat) | agent v0.94.0→**v0.95.0**, controller v0.169.0→**v0.171.0**, hub v0.73.1 | **PROVEN-LIVE (healthy path + delivery)** | **2026-07-25 (v0.95.0 + v0.171.0 — the SMART-coverage fix):** the card on guest 9201 now shows BOTH real disks with **real verdicts + human model labels** — **„AirDisk 512GB SSD" → Rendben (34°C)** (the system SSD, via LVM/dm resolution) and **„TOSHIBA MQ04ABF100" → Rendben (30°C)** (the USB, via union-path SMART). `/disks` carries `smart.health=PASSED` + `model_name` for both. This reverses the 2026-07-24 „Nincs adat on a raw UUID" state (`SPIKE-smart-coverage-2026-07-25.md` had proven both disks answer `smartctl -a -j` PASSED but the agent never asked). Prior: verdict table (+≥90 red-proof); check first-run/degradation/recovery/UNKNOWN tests; hub allowlist test. **Notification pipeline PROVEN-LIVE 2026-07-24** — a `disk_health_degraded` POST (the exact `notify.PushEvent` wire call) was **400-rejected by hub v0.73.0** and **200-accepted + „Operator email sent" by hub v0.73.1** | No new smartctl load; feature-detect by payload presence → **MinAgent floor unchanged**; no sudoers/`-d sat` change. **No global banner** (deliberate). Agent v0.95.0 fixes: union-path SMART (Fix B) + LVM/dm whole-disk resolution incl. the builtin `local` on the LVM root (Fix A, SMART-only — never touches backing/durable_id) + `model_name` capture. **A live FAILING/Figyelmeztetés verdict still needs a genuinely degrading disk** (the demo disks are healthy) — that arm stays unit+red-proof covered. **SMART history/trending (hub-side) PARKED** (ROADMAP R-73) |
+| Lemez-egészség felügyelet: per-disk SMART kártya („Lemezek állapota") + degradáció-riasztás (Rendben/Figyelmeztetés/Hiba/Nincs adat) | agent v0.94.0→**v0.95.0**, controller v0.169.0→v0.171.0→**v0.215.0**, hub v0.73.1 | **PROVEN-LIVE (healthy path + delivery + the severity wire).** **IMPLEMENTED, NOT proven-live: the Hiba-from-counters path** (v0.215.0) — it has never fired on real hardware, only against the committed fixture's values in unit tests (**R-332**) | **2026-07-25 (v0.95.0 + v0.171.0 — the SMART-coverage fix):** the card on guest 9201 now shows BOTH real disks with **real verdicts + human model labels** — **„AirDisk 512GB SSD" → Rendben (34°C)** (the system SSD, via LVM/dm resolution) and **„TOSHIBA MQ04ABF100" → Rendben (30°C)** (the USB, via union-path SMART). `/disks` carries `smart.health=PASSED` + `model_name` for both. This reverses the 2026-07-24 „Nincs adat on a raw UUID" state (`SPIKE-smart-coverage-2026-07-25.md` had proven both disks answer `smartctl -a -j` PASSED but the agent never asked). Prior: verdict table (+≥90 red-proof); check first-run/degradation/recovery/UNKNOWN tests; hub allowlist test. **Notification pipeline PROVEN-LIVE 2026-07-24** — a `disk_health_degraded` POST (the exact `notify.PushEvent` wire call) was **400-rejected by hub v0.73.0** and **200-accepted + „Operator email sent" by hub v0.73.1** | No new smartctl load; feature-detect by payload presence → **MinAgent floor unchanged**; no sudoers/`-d sat` change. **No global banner** (deliberate). Agent v0.95.0 fixes: union-path SMART (Fix B) + LVM/dm whole-disk resolution incl. the builtin `local` on the LVM root (Fix A, SMART-only — never touches backing/durable_id) + `model_name` capture. **2026-08-14 — a genuinely failing disk HAS now been seen, and it broke three assumptions** (`audits/DIAG-smart-passed-trap-2026-08-14.md` + two committed fixtures: raw `smartctl -a -j` and 406 `smartd` lines from ST3000VX010 S/N Z6A07P2G). **(1)** `smart_status.passed` is STRUCTURALLY incapable of failing on unreadable sectors — attrs 187/197/198 all carry `thresh: 0` and a normalized value floors at 1 — so the drive read PASSED at 352 pending sectors and 1001 uncorrectable reads. **(2)** The alert it did produce carried severity `"warn"`, which the hub coerces to `info` and never emails: **the counterfactual is ZERO emails about this drive** (R-328, fixed controller v0.215.0, and the `warning`-vs-`warn` pair proven side by side in `notification_log` on 2026-08-14 — `sent` vs no row at all). **(3)** The old check spoke once and forgot on restart, so between 8 and 352 sectors it emitted nothing. v0.215.0 adds the sustained/count/heat Hiba rules, persisted state and an hourly cadence. **The verdict half of that arm remains unit+red-proof covered only** — no live drive has reached Hiba from counters (R-332). **SMART history/trending (hub-side) PARKED** (ROADMAP R-73) |
 | App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **IMPLEMENTED** | controller v0.120.0 (dead-app alerting, `app_start_failed`, one-event-per-transition red-proofs); `CAMPAIGN-3` F11 surfaced the gap | End-to-end crash→customer-email delivery never live-confirmed (6B deferred / 6C inconclusive: clean stop ≠ crash); anti-spam unit-proven |
 | Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **IMPLEMENTED** | feature long-standing; config page renders (`CAMPAIGN-2` T-PAGE-ALL is GET-only) | The config-save+restart flow is exercised in no campaign (CAMPAIGN-3 explicitly skipped interactive app config). Demoted: T-PAGE-ALL is a page-render smoke test, not this flow |
 | Backup classification: 13 bind-bearing apps carry mandatory/optional/excluded classes | catalog, controller v0.132–133 | **PROVEN-LIVE** | `SPIKE-backup-classification-2026-07-14`, `CAMPAIGN-6D/6E` | Remaining ~39 apps are legacy-class by design (unit-only offsite) |
@@ -71,15 +122,16 @@
 > about the **failure** and the row is about the **mechanism**; that is not a contradiction, and both
 > cite the same evidence.
 >
-> The matrix's blank RTO/RPO cells are deliberate: no number is estimated anywhere. Two counts of
-> Tier-2 app coverage disagree (9/43/1 vs 7/45/1) and are **both** recorded there, unresolved —
-> do not adopt either from this page.
+> The matrix's blank RTO/RPO cells are deliberate: no number is estimated anywhere. The two counts of
+> Tier-2 app coverage that used to disagree (9/43/1 vs 7/45/1) were **settled 2026-08-31 by measurement
+> at catalogue `459766cb1639`: A = 7 · B = 45 · C = 1** (`07-backup-architecture.md` §6.2, which also
+> records which prior count was wrong and why). Take the number from there, not from memory.
 
 | Scenario | Components | Status | Evidence | Gap / roadmap |
 |---|---|---|---|---|
 | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded **DEGRADED** rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | `E2D-fresh-vm-2026-07-29` C1 (real 1.22.0 install, rc=0, `Day-0 provision SUCCESS`) + C2 (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort) | Case A (a second drive already present at install) has never fired naturally — only Case B has |
 | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 **DB replay route → `07-backup-architecture.md` §8 row 3** |
-| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | **Route + RTO → `07-backup-architecture.md` §8 rows 1, 2, 4, 5.** The matrix records that the copy's `recovery-unit/` mirror is read by no path (→ R-102) |
+| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | **Route + RTO → `07-backup-architecture.md` §8 rows 1, 2, 3b, 4, 5.** **R-403 (controller v0.230.0, 2026-08-31) changes NO status on this row and that is stated rather than left ambiguous:** the nightly copy now refuses to replace a complete unit package with an empty one (`07` §8.2), which removes a way the route could be DESTROYED between uses — it does not change what the route can be relied on for, and every leg it covers is the same one it covered yesterday. Proven live on demo-hp: the same state that deleted 120 082 104 B on v0.229.0 preserved all 7 files on v0.230.0. **R-102 CLOSED 2026-08-31 (controller v0.229.0):** the copy's `recovery-unit/` mirror is now restorable — „Teljes visszaállítás a másolatból" / `POST /backup/tier2/unit-restore` — and was proven live on `demo-hp` with the primary unit moved aside (`audits/DRILL-r102-tier2-unit-2026-08-31/`, §8 row 3b, 28.65 s). R-103 closed with it: the refusal that used to name a button on another page now offers the action on the row itself |
 | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect). **2026-08-04 (R-193/R-197, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`) — the row is NOT overclaiming and the guard is not the gap; the CADENCE is.** This row already recorded that a recreated data volume orphans the repo, and the spike confirms the mechanism at source: `WriteOffboxSecrets` (`offbox.go:392`) mints a fresh 256-bit repo password whenever `<DataDir>/offbox/repo_password` is absent, and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly one caller, a web form a human pastes into. What was NOT recorded is that this now fires on an **ordinary, planned, unattended guest rebuild**, on every box: measured on BOTH demo boxes 2026-08-03/04 by comparing `host_escrow.restic_pw_sha256` against `host_escrow_superseded.restic_pw_sha256` (demo-hp `8e03eddf…`→`8a9e33aa…`, demo-felhom `48741892…`→`c60c8bc7…`), orphaning **15 snapshots / 40.9 MB** and **36 snapshots / 1.14 GB** respectively. **demo-felhom is the important half:** it kept its DELIVERY (a stale staged secret restored the target in 76 s) and lost its REPOSITORY anyway, with **nothing marking the escrow stale for 13 h** — `escrow_stale` is wired to `ReissueCredentials`, the one path that does NOT change the repo password (**R-196**), and absent from the rebuild path that does. Status unchanged: the classify-and-move-aside guard remains PROVEN-LIVE and correct, and is predicted (not yet measured) to refuse the 2026-08-05 run on both boxes rather than start a silent fresh history |
 | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** (2026-07-20) | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* **2026-07-20: the clean run HAPPENED** — endpoint-level supervised reconstitute of immich from snapshot `49e7cb46` (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no `already exists`, operation reported SUCCESS, immich's own DatabaseService logged `No schema drift detected` twice, 11 assets `active`, 4/4 containers healthy, 231 `public` indexes. **Operator confirmed the immich timeline renders correctly after the reconstitute** (screenshot held, 2026-07-20). Evidence: `felhom-controller/REPORT.md` §4b. **2026-07-20, LATER THE SAME DAY — the destructive drill RAN and the row now earns PROVEN-LIVE.** The operator deleted the photos in immich own UI **and emptied the trash** (the step whose absence makes a drill prove nothing — the round-1 lesson), then restored through the customer-facing UI. **`40 file(s) placed`** against the 6 of the earlier non-destructive run — the files were really gone and really came back — plus 1 DB dump replayed rc-0, 11 assets `active`, `No schema drift detected`, timeline confirmed by the operator. This is the destroy-then-recover proof the 6D downgrade asked for, and it was taken through the customer own buttons, not endpoint shortcuts. Evidence: `felhom-controller/REPORT.md` 4e **Route + RTO → `07-backup-architecture.md` §8 rows 3, 4** — the matrix also records that no offsite action unpacks the named-volume tars it captures (→ R-107) |
 | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
@@ -104,7 +156,10 @@
 | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-5 (scope=app round-trip, byte-identical); `storage-lifecycle-acceptance-2026-06-15` (two migrate-all runs via dashboard UI, sha256 byte-identical) | (Cited `CAMPAIGN-2` T-STG-MIGRATE-* were auth-hollow.) "crash-safe" is design-level (copy→verify→remove) — no clean live crash-during-migration PASS |
 | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | `SPIKE-nas-verify-2026-07-11`, `SPIKE-nas-dsm-2026-07-11`, `CAMPAIGN-3-2026-07-11` (boot/reassert fixes) | |
 | **Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace** | controller v0.187.0 | **PROVEN-LIVE** (2026-07-30) | `audits/R108-network-app-namespace-2026-07-30.md`. Same-box before/after on demo-felhom through the exact endpoint the UI invokes (`POST /api/storage/migrate-app`, authenticated + CSRF): **pre-fix v0.186.0** the target was never examined — both a NAS-shaped and an unregistered path passed straight into `MigrateApp` and failed only on the app name (409); **post-fix v0.187.0** both are refused 400 with a Hungarian reason, while a real local drive still reaches `MigrateApp` (409) proving the guard is not over-broad. On demo-hp (the box with a REGISTERED share) the network-specific refusal fires. Non-effect verified in the registry: no `migrated_to`, nothing decommissioned, app `HDD_PATH` unchanged, no `backups/` on the share | **Closes R-108 and UNBLOCKS D5** (`07-backup-architecture.md` §7.3, §10.1). The share-root `:rslave` FileBrowser bind is deliberately UNCHANGED — load-bearing for automount wake, and unscopable — verified byte-identical by diffing demo-hp's generated compose before/after. Fails closed: `/mnt/felhom-drives` holds both kinds, so an unregistered path under it is un-classifiable and refused. **Not exercised:** the deploy POST and decommission-migrate refusals are unit-tested (non-effect, nil `stackMgr`) but were NOT live-fired — only migrate-app was. `.fab`-export-onto-NAS remains open (→ R-126) |
-| **A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5)** | controller v0.188.0 | **PROVEN-LIVE** (2026-07-30) | `audits/D5-drive-alone-restore-2026-07-30.md`, `07-backup-architecture.md` §7.4. On a scratch drill guest on felhom-pve, through the real endpoints (`POST /api/stacks/{app}/deploy` → `POST /api/backup/run` → `POST /backup/restore`): AdventureLog (`SECRET_KEY` **data_key** + `DB_PASSWORD`) restored with the guest's `app.yaml` **moved aside** → `secrets recovered=2/2`, **27.6 s**, `Restore-from-unit completed`. **The observable is the DATA, not the exit code:** the app itself then read the seeded customer row **over TCP with its own credential** (`connected_as=adventurelog over_TCP=True`), 51 Django tables intact, and the discriminator held — the pre-backup row returned while a row added AFTER the backup was gone, so the volume tar was genuinely restored. The unit held **no `.sql` dump**, so the DB came back from the volume tar, which is exactly the case a regenerated password would have broken. **The withheld half is proven too:** Grafana's `type: password` admin login was live in its container and `ENC:` in the guest, yet appeared in **0 files** anywhere under the backup namespace, and the unit's app.yaml header names it as withheld | **Ruling (operator, 2026-07-30): `type: secret` travels, `type: password` NEVER does, minus the `nonPortableSecrets` code register (`vaultwarden/ADMIN_TOKEN`).** Plaintext on the drive, like the data — defensible **only because** the internet-reachable class is withheld; the two are coupled and must not be relaxed independently. **The brief's own proposal (data_key-only) was tested and rejected:** the flag is unreliable (4 encryption keys the catalog itself labels as such are unflagged → **R-127**) and a DB password is not resettable in practice (`POSTGRES_PASSWORD` is ignored once PGDATA is non-empty). Precedence: **the unit wins** over the guest, because the unit's secrets match the data being restored. The fail-closed data-key gate is UNCHANGED. **Not exercised live:** the withheld-class O4 regeneration on restore, and Tier-2's own cross-drive copy of a secret-bearing unit (both unit-tested only). Venue was a **fixture-class scratch guest**, correct per `runbooks/target-selection.md` since D5's claim is about restore CODE, not the install path |
+| **A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5)** | controller v0.188.0 | **PROVEN-LIVE** (2026-07-30) | `audits/D5-drive-alone-restore-2026-07-30.md`, `07-backup-architecture.md` §7.4. On a scratch drill guest on felhom-pve, through the real endpoints (`POST /api/stacks/{app}/deploy` → `POST /api/backup/run` → `POST /backup/restore`): AdventureLog (`SECRET_KEY` **data_key** + `DB_PASSWORD`) restored with the guest's `app.yaml` **moved aside** → `secrets recovered=2/2`, **27.6 s**, `Restore-from-unit completed`. **The observable is the DATA, not the exit code:** the app itself then read the seeded customer row **over TCP with its own credential** (`connected_as=adventurelog over_TCP=True`), 51 Django tables intact, and the discriminator held — the pre-backup row returned while a row added AFTER the backup was gone, so the volume tar was genuinely restored. The unit held **no `.sql` dump**, so the DB came back from the volume tar, which is exactly the case a regenerated password would have broken. **The withheld half is proven too:** Grafana's `type: password` admin login was live in its container and `ENC:` in the guest, yet appeared in **0 files** anywhere under the backup namespace, and the unit's app.yaml header names it as withheld | **Ruling (operator, 2026-07-30): `type: secret` travels, `type: password` NEVER does, minus the `nonPortableSecrets` code register (`vaultwarden/ADMIN_TOKEN`).** Plaintext on the drive, like the data — defensible **only because** the internet-reachable class is withheld; the two are coupled and must not be relaxed independently. **The brief's own proposal (data_key-only) was tested and rejected:** the flag is unreliable (4 encryption keys the catalog itself labels as such are unflagged → **R-127**) and a DB password is not resettable in practice (`POSTGRES_PASSWORD` is ignored once PGDATA is non-empty). Precedence: **the unit wins** over the guest, because the unit's secrets match the data being restored. The fail-closed data-key gate is UNCHANGED. **Not exercised live:** the withheld-class O4 regeneration on restore (unit-tested only). ~~Tier-2's own cross-drive copy of a secret-bearing unit~~ — **EXERCISED LIVE 2026-08-31 (controller v0.229.0):** docmost restored from `/mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit` with the guest's `app.yaml` **moved aside** AND the primary unit moved aside, `secrets recovered=2/2` (`APP_SECRET`, `DB_PASSWORD`) taken from the MIRRORED unit's `compose/app.yaml`; the guest's `app.yaml` was rebuilt from it at 0600 and the app then read its own rows over TCP with its own credential. Evidence: `audits/DRILL-r102-tier2-unit-2026-08-31/phase8-scenarioD-guest-appyaml-aside.log`. Venue was a **fixture-class scratch guest**, correct per `runbooks/target-selection.md` since D5's claim is about restore CODE, not the install path |
+| **A restore SAYS what it returned, and refuses what it cannot do** — the four restore-surface truth defects from the 2026-08-21 drill | controller **v0.226.0** (R-353, R-357, R-358, R-360, R-396) | **PROVEN-LIVE (2026-08-30) for three of the four; R-357 is IMPLEMENTED only** | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt`, controller `CHANGELOG.md` v0.226.0 + `REPORT.md`. Driven on `demo-hp` through the endpoints the UI invokes (no browser on DooPlex; the residual is client-side rendering). **R-353:** the sentence read off the customer's own wizard page — `A(z) opengist: 1 adatkötet visszaállítva — az alkalmazás újraindult.` with real counts (1 volume of 1 listed, 0 databases of 0 listed, and correctly no database clause). **R-360:** in the exact flag state that produced the bug (display flag true, concurrency flag false) the delete was refused and **a planted canary file survived**. **R-358/R-396:** a `mode=unit` restore wrote `{"schema":1,…,"full":false}` at mode 0600 with no `.tmp` left, and the gate logged `scratch holds a UNIT-ONLY restore … place-to-live stays closed` | **WHAT IS AND IS NOT CLAIMED, split deliberately.** **R-357 (the destructive restore's free-space gate) is IMPLEMENTED, NOT PROVEN-LIVE** — filling a real filesystem is a drill step, not a build step, so it rests on seam tests (`SetOffboxFreeFn`, `SetOffboxSizer`, and the new `SetOffboxLatestSnapshotFn`) whose central assertion is that `StopStack` was never called. **R-353's Scenario B — the "backup held only settings" sentence — was NOT reproduced live either**, and the reason is stated rather than glossed: no app on `demo-hp` still has a data-less unit (the drill's opengist has been recaptured and now lists one volume dump), and falsifying a manifest to produce it is the hand-set-state shortcut this project forbids. That branch is unit-proven only. **This row is about the MESSAGE and the REFUSALS, not the recovery mechanism** — `07-backup-architecture.md` §8 row 3 keeps its PROVEN status because the restore always did return what the unit held; what it could not do was say so |
+| **The box PROVES its own off-site copy still HOLDS something — a backup that is intact and EMPTY is caught without a person** | controller **v0.231.0** (R-87) + hub **v0.110.0** | **PROVEN-LIVE (2026-08-31) for the judgement, the alarm, the cleanup, the rotation, the read-only guarantee and the skip-if-busy hazard control; the SCHEDULED FIRING is IMPLEMENTED only** | `documentation/tests/r87-offsite-proof-2026-08-31/`. Driven on `demo-hp` through the endpoint the debug button invokes. **The failing case was produced and caught:** a hollow unit — compose declaring `opengist_data`, manifest declaring nothing — was pushed to the live store, and the proof returned `verdict:"fail"` with `volumes_expected_none_captured: opengist_data`, emitted **exactly one** `offsite_proof_empty` at severity `error`, accepted by the hub **HTTP 200** (which is itself the proof the allowlist entry landed — an unallowlisted type is 400'd and vanishes), and deleted its scratch. **The passing case was proven five times over** (bookstack, calibre-web, docmost, kimai, opengist), 2.2–4.0 s each, and the customer's own verification copies were untouched throughout. **The read-only guarantee was measured with a positively-controlled lock sampler** — it saw a lock appear and vanish across a real `restic check`, and **zero** across the proof, including a direct 6× test of the snapshot-lookup argv. **The skip-if-busy control fired live and unplanned:** a proof launched while the off-site backup run held the flag returned `skipped:true, duration_ms:0` with no verdict and no alarm | **⚠ WHAT A PASS MEANS, AND WHAT IT DOES NOT.** It means the newest off-site snapshot of ONE app contains what that app is supposed to have — judged from the unit's own captured compose, not from the live box. **It does NOT mean a restore puts data back into a running app**: the proof restores to a throwaway folder, looks, and deletes, and `07` §8 matrix row 4 is deliberately NOT moved. **It also does not vouch for the BYTES** — nothing available can: restic 0.14.0's `restore --verify` passed a byte-level corruption with size and mtime preserved (measured, 131 ms on a 213 MB tree), and the unit manifest hashes 4 918 B of a 213 231 242 B unit (R-409). **2026-09-01 (R-414, controller v0.232.0): it can now run on a box with NO registered data drive.** The first unattended firing, on `demo-felhom`, REFUSED — *"nowhere to restore to"* — because that box has `storage_paths: []` and the scratch resolver never consulted the system data path. A **unit-only** restore now falls back there (where a driveless app's unit already lives, `07` §7); a **full** restore still refuses, because the SSD is a state-only tier. And a proof that cannot start now records `cannot_run` instead of nothing, so `last_proof_result` is never ABSENT — absent already means *a controller too old to have the feature*. **PROVEN LIVE on `demo-felhom` 2026-09-01:** `verdict:"pass"` on `61e9cf30` in 2.117 s, recorded, and the scratch deleted. **The nightly firing IS now proven** — it ran unattended on `demo-hp` at 05:30 on 2026-09-01 (`bentopdf PASSED on 9d002b38 in 2.315s`), which this row previously listed as implemented-only. The job is confirmed REGISTERED on `demo-hp` (`Daily job offsite-proof scheduled for 2026-09-01 05:30 CEST`), which is not the same claim, and the fleet is on 0.230.0 until a golden carries 0.231.0 |
+| **The off-site store is VERIFIED on a cadence — something checks that the customer's backups are still readable** | controller **v0.228.0** (R-359, R-397, R-399) | **PROVEN-LIVE (2026-08-30, re-proven at FULL DEPTH 2026-08-31) for the check, the notifier and the hazard control; the SCHEDULED FIRING is IMPLEMENTED only** | `documentation/tests/r359-integrity-2026-08-30/`. Driven on `demo-hp` through the endpoint the debug button invokes. A throwaway repo was built, checked healthy (**negative control first**), then one pack corrupted; the live store was checked read-only in **35.0 s**; and the notifier fired end to end — `Event pushed: backup_integrity_ok (info)`. The hazard control was observed live: a second check fired while the first held the single-writer flag returned `skipped:true, duration_ms:0` — **it never ran restic at all** | **⚠ WHAT AN `ok` MEANS — CHANGED 2026-08-31 (R-399, controller v0.228.0): the check now RE-READS THE DATA.** The default is `--read-data-subset=100%`, so an `ok` means every stored byte was downloaded and re-hashed, not merely that the catalogue hangs together. **The reason is measured, and it is why the default must not be turned back down to save four seconds:** a pack corrupted WITHOUT a size change made a structure check return `no errors were found`, exit 0, while every `--read-data*` form caught it. Cost curve on 134.3 MB: structure 35.0 s · 10% 35.9 s · 50% 37.3 s · 100% 39.2 s — **and those do NOT extrapolate**, which is why v0.228.0 ships a slow-check WARN (R-401) rather than a rotation schedule. `off` returns a box to structure depth. **PROVEN-LIVE at the new depth 2026-08-31 on `demo-hp`**, endpoint-level, with the restic argv observed from the guest: default → `… check --read-data-subset=100%`, 38.7 s; `off` → `… check`, 34.7 s. **The weekly firing at the new depth is IMPLEMENTED only** — the job is confirmed REGISTERED on BOTH demo boxes (`Daily job offsite-integrity scheduled for 2026-09-01 06:00 CEST`), which is not the same claim. `demo-felhom` reached 0.228.0 by SELF-UPDATE on the 2026-08-31 floor raise and re-registered the job itself, so the depth change is on the fleet and not only on the box that was deployed to by hand. **This is a readability check and NOT a restore-test** — R-87 remains open and the two are routinely conflated because their register rows are adjacent |
 | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E4 (yanked-while-running → agent auto-rebind) + E10 (re-enroll, data intact); `CAMPAIGN-4`/`6A` (3 USB re-establish across device-letter reshuffle) | (Cited `RUNBOOK-usb` could NOT complete a wizard enrollment; `CAMPAIGN-2` legs were auth-hollow.) Fresh-USB **wizard enrollment** specifically still unproven |
 | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E9 (decommission-anyway → bind detached, parent mp untouched, reboot-safe) + E12 (eject drive holding all apps) | (Cited `CAMPAIGN-2` T-STG-DECOM-* were auth-hollow; `SPIKE-decommission` was report-only, button still vestigial.) |
 | Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | `CAMPAIGN-4-2026-07-13` (F12 fix HOLDS: demo-host reboot + 5-boot storm, 0 ordering cycles, caps 63/63, WG re-handshake) + `CAMPAIGN-6A-2026-07-14` 1D (re-arm reboot-survival across 9 guest + 1 host reboots) | (`CAMPAIGN-3` F10/F11/F12 were the CRITICAL/HIGH *failures*; fixes shipped in agent v0.85 and were re-validated live in 4/6A — cite the validation, not the finding.) Residual: `skip-active` on `pct reboot` carried by the heal path; a NAS outage spanning a guest reboot can strand the share until agent restart (6A) |
@@ -126,6 +181,7 @@
 | Multiple household users / per-person accounts | — | **MISSING** | — | Single dashboard password; acceptable for alpha → R-15 |
 | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | `SPIKE-oob-wg-operator-peer-2026-07-05`, `SPIKE-felhom-sshd-2026-07-05` | Mutual-repair desired-state arc not built → R-13. **CHECKED 2026-08-08 (R-260) and this row was NOT claiming something untrue** — it claims the capability is implemented, never that it is monitored, so no correction was owed. What WAS untrue is narrower and sat one layer down: **the hub's own OOB health check could not see whether the operator's key was installed.** `HostOOBRow` mirrored five of the agent's eight OOB fields, so `operator_key_configured` — emitted every heartbeat since agent v0.72.0, i.e. from this row's own vintage — was discarded by `encoding/json` on arrival, and `oobDegraded` returned `ok` for a box with felhom-sshd active, reachable, a valid config, a configured peer and **no operator key at all**. `operator_peer_configured`, which it did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Fixed hub v0.99.0; the missing key now degrades and the alert NAMES it; a stanza too old to carry the field is reported distinctly and is never a silent ok. Pinned end-to-end from raw report JSON by `TestHostOOB_MissingOperatorKey_EndToEnd` and `TestHostOOB_NoKeyField_IsNotSilentlyOK_EndToEnd` |
 | The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent **v0.119.0**, hub **v0.85.0** | **PROVEN-LIVE** (2026-07-31) | `audits/host-addresses-visible-2026-07-31.md` | Before this the LAN IP was **not reportable at all** — `HostMetrics` carried no address of any kind — and the WG IP existed only in `/offsite`'s peer table keyed by pubkey (peer→host, never host→peer). New wire field `addresses[]`, one row per (interface, address); `IsGlobalUnicast()` is the whole filter, chosen by MEASURING both demo boxes, and it needs no veth/fwbr denylist because that plumbing carries no IP. Rendered live on both 0.119.0 hosts matching their `ip addr` ground truth exactly. **Two honesty properties carry the risk and are both red-proofed:** WireGuard shows the hub ALLOCATION and whether the box CONFIRMS holding it (allocation alone cannot tell a live tunnel from a peer never applied), and an agent below 0.119.0 renders **UNKNOWN, never "no addresses"** — proven live on `drill-r50-0a4f9a` (0.113.0). **Not covered:** a two-LAN-bridge box and a real WG drift, neither of which exists to observe |
+| The operator can see whether a managed host's **guests still have working networking** — and **how often the watchdog had to repair them** | agent **v0.92.0** (emitter, 2026-07-21), hub **v0.104.0** (reader, 2026-08-13) | **IMPLEMENTED** | `backlog/OPEN-ITEMS.md` R-319; `hub/internal/web/hosts_guestnet_test.go` (7 tests, fixtures copied verbatim from `demo-felhom-8363b5`'s live `host_reports` row) | The agent emitted `guest_net` on every heartbeat for **twenty-three days** while the string occurred **nowhere** in `felhom.eu/hub/` — stored as raw text in `report_json`, read by nothing (R-260/R-264, the first of that census's readers to be built). **The fact that carries the risk is `heals_last_hour`, not `state`:** a guest the watchdog keeps repairing is healthy at every instant anyone looks, so rendering the state alone would give it a green tick — the failed-disk-drawn-as-a-healthy-empty-disk shape. `heal_succeeded` is decoded beside it, because six FAILED repairs is a guest that is down while six successful ones is a nuisance. **Unknown is never drawn as healthy:** three absences, three sentences (agent < 0.92.0; a capable agent that sent nothing; a guest whose own state the watchdog did not assert), and a malformed stanza degrades to unknown without a 500. **Three red-proofs, each mutation asserted applied by grep before its run**, including the one that matters — removing the unknown branches and watching a silent machine render as healthy. **Positive control that it is WIRED and not merely written: the wire-contract gate's checked-tag count rose 182 → 190** as the eight `guest_net` allowlist entries were deleted (an allowlisted tag is skipped, so leaving them would have meant these fields were never checked) | **IMPLEMENTED, not PROVEN-LIVE, and the distinction is the honest half.** Every scenario is proven against the real wire in tests, and the healthy case renders correctly for the live fleet — but **no machine has ever been observed with a climbing repair count on this card**, because neither demo box has needed a repair since the watchdog shipped. The signal this card exists for has therefore never been seen firing on hardware. It moves to PROVEN-LIVE the first time a real repair count is watched appearing. **No alarm was added, deliberately** (R-319): the incident behind this was about nobody being able to SEE the condition, and a new email on a fleet of two demo machines is untested noise — revisit when a third machine exists or when a count is seen climbing |
 | Break-glass management-plane recovery | agent v0.71, hub v0.84 | **IMPLEMENTED** | `runbooks/break-glass.md` | hub v0.84.0 adds an **operator-SESSION** retrieval path (host page → Console access → Reveal; `POST /hosts/{id}/reveal-recovery-credential`, CSRF-gated, writes a customer-visible `recovery_credential_revealed` event) beside the pre-existing **global-key** one (`GET /api/v1/admin/hosts/{id}/recovery-credential`), which is untouched and stays the route for when the hub UI itself is down. **The credential half is now PROVEN (2026-07-31):** the vaulted `demo-hp-bb76ea` password was verified against the box's own `/etc/shadow` hash AND minted a real PVE ticket — `POST /api2/json/access/ticket` → **HTTP 200, `root@pam`, 367-char ticket**, the exact API the login form submits to. Still IMPLEMENTED rather than PROVEN-LIVE overall, because the path has not been exercised on a REAL lockout (SSH was available throughout). Discovered during that check: the card's Copy button shipped `disabled` until a Reveal and silently no-opped, leaving ANOTHER host's password in the clipboard — fixed in hub v0.86.0. The vaulted secret is plaintext at rest → **R-133** |
 
 ## F. Notifications & monitoring
@@ -156,5 +212,7 @@
 | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | `OffsiteChecker` (`hub/internal/monitor/offsite.go`): fill 90/95% vs soft quota, staleness >48h | No live-fired leg: `CAMPAIGN-offsite-overnight-2026-07-10` recorded no quota/fill/staleness emails, and the freeze write-block was **inconclusive** (only the Hetzner `readonly:true` API op succeeded). Demoted |
 | **Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)** | hub v0.64.0 (R-5) | **IMPLEMENTED** (data pipeline PROVEN-LIVE) | `monitor.OffsiteBoxChecker` — fetch-throttled Hetzner GET (1/15 min), fill (used/`storage_box_type.size`, 80/90%) + oversubscription (Σ shared+enabled ConfigJSON quotas / capacity, 2.0×), escalation-only operator alert on the customer-less `"pool-box"` scope; Offsite-tab panel + dashboard tile. **Phase-0-pinned** live shape (box 611714) + **live-computed in-cluster:** `0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x`. Tests + 4 red-proofs; hub v0.64.0 REPORT | Two open legs: the **UI render** is unit-verified only (hub UI password-gated → no screenshot); the **alert emails** are unit + red-proof verified but NOT fired live (real pool nominal — a live-fire emails Viktor). Thresholds pending Viktor's ruling (named config keys). READ-ONLY (GET) |
 | **Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge)** | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** (data pipeline PROVEN-LIVE) | The PBS DR datastore (`felhom-offsite` on ep0) fill — NOT a Hetzner box. **Option A:** a read-only `usage` op on the `felhom-tenantsync` ep0 forced command (twin of `fingerprint`; `df` on the datastore path — no customer_id, no admin token, NO mutation), polled by `monitor.PBSDRBoxChecker` (OffsiteBoxChecker clone; 15-min throttle; states ok/unavailable/degraded; fill 80/90% on the `"pbsdr-box"` operator scope). `/offsite` split into Restic + PBS DR tabs; two dashboard gauges. **Graceful: hub deploy ⟂ ep0 update** (ep0 ≤ v1.1.0 → gauge "n/a" until updated). **Phase-0-pinned** (`df` on ep0 PBS 4.2.3) + **live-computed in-cluster** (ep0 updated to v1.2.0 this session): `19.1% full (7.1 GB of 37.2 GB)`. 10 Go tests + a bash harness + 3 red-proofs; hub v0.65.0 REPORT | Open legs: **UI render** unit-verified only (hub UI password-gated); the **fill alert email** is unit + red-proof verified, NOT fired live (datastore nominal at 19%). Separate PBS threshold keys (default 80/90); no oversubscription (namespaces, not quotas). READ-ONLY |
+| **The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)** | hub **v0.106.0** (R-339) | **IMPLEMENTED — deliberately NOT proven-live** | Both box checkers count consecutive failed fetch windows and emit `pbsdr_box_unreachable` / `offsite_box_unreachable` (severity `warning`) past a default 3 windows (≈30–45 min), each with a paired `*_recovered` all-clear routed via `recoveredPairedDownTypes` — required because the recoveries are severity `info`, which `severityNotifies` drops. Scopes stay customer-less (`pbsdr-box` / `pool-box`) → operator channel only. Fill logic untouched: a degraded read still drives no band transition. Evidence: `internal/monitor/box_reachability_test.go` + the cross-package wiring test in `internal/notify/`, which asserts an actual operator mail rather than a map entry. **Filed BECAUSE of a measured gap**, not a hypothesis: the 2026-08-18 ep0 outage ran 9 h 37 m with the hub silent | **The gap that remains is R-340**, and it is not small: the ep0 read is the `usage` op, which rides the LOCAL API daemon — the daemon that incident explicitly cleared — so this check would have shown GREEN for that entire outage. It closes "ep0 is unreachable as a host"; it does not close what actually happened. **No live or constructed outage has exercised the emit path**, and one cannot be manufactured against ep0 (Tier 2, protected) |
 | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | 07-13 closing bundle | |
-| Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | |
+| Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | |
+| **A deletion of a customer's off-site history is NOTICED within a day** | hub **v0.111.0** (R-431) | **IMPLEMENTED — not yet PROVEN-LIVE** | 09-01 | `hub/internal/monitor/offsite.go` — third signal beside FILL and STALENESS. **On the hub deliberately:** a detector on the box is one the deletion can silence. Alarms when the reported count falls by more than HALF and by at least 5, guarded by `StatsKnown` (R-331), the declared `State` (R-204) and run success (R-100). **Threshold reasoned, not invented:** over 12 898 reports every decrease lands on ZERO and predates `stats_known`; in the 380-report `stats_known` window there are none. **ACCEPTANCE: 9 009 real points replayed → ZERO alarms** (`offsite_r431_test.go`, fixture committed). **What PROVEN-LIVE would need and this does NOT have:** a real drop observed on a live box producing a real mail — the live firing done at ship time was driven through the hub's own path with synthetic counts, which is an end-to-end delivery proof, not a proof that a genuine deletion is caught. |

documentation/architecture/01-topology-and-trust.md

@@ -1,5 +1,20 @@
 # Felhom Controller Architecture — Part 1: Topology & Trust
 
+> **How to read this document.** Two kinds of statement appear, and where this document marks them it
+> marks them like this — the same wording as `07-backup-architecture.md:11-17`, carried here on
+> 2026-08-22 (R-376) so a reader meets one convention and not eight:
+>
+> - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet.
+> - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation.
+>
+> **Statements in this document are NOT yet all marked.** Marking them wholesale is a large judgement
+> exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376).
+> **An unmarked statement therefore means "not yet classified", never "observed".** That ambiguity is
+> exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat
+> unmarked beside a marked `[FACT]`, and was read as an observation and reported as a defect.
+
+
+
 **Status:** draft (decisions from the topology/trust design sessions).
 **Platform facts** referenced here live in `docs/proxmox-platform.md`; this document
 records *Felhom's decisions*, not Proxmox behaviour.
@@ -147,9 +162,23 @@ credentials.
   at attach), role, encrypted credentials, schedule/retention. The agent creates the Proxmox
   storages, continuously checks presence/reachability, and reports per-target status (a
   disconnected target → actionable notification).
-- **App data placement is per-volume, not per-app:** `.felhom.yml` classifies each volume
+- **[DESIGN] App data placement is per-volume, not per-app:** `.felhom.yml` classifies each volume
   **hot** (DB/config/cache → fast storage, enforced) vs **bulk** (media/files → may be slow).
   A photo app's DB stays on SSD while its blobs go to the USB.
+
+  > **Marked [DESIGN] on 2026-08-22 (R-376), and the pointer is honest about what it can point at.**
+  > **This decision was never recorded as a decision anywhere** — it was established by reading, not
+  > by citation: it exists as this bullet and nowhere else, with no dated entry in the decision log
+  > and no `R-` row. A log entry was written on 2026-08-22 (`CONTEXT.md`, "App data placement is a
+  > DECISION") **to give it a home, not to claim it was decided then**; the choice is older than the
+  > entry and its original date is not on record.
+  >
+  > **What being unmarked cost.** The consequence of this bullet — that 40 of 53 catalogue templates
+  > declare no configurable path because they are all-hot — is stated as **[FACT]** at
+  > `07-backup-architecture.md:296-299`. A reader met a marked observation beside an unmarked choice
+  > and reasonably asked whether it *should* be so. **Between 19 and 22 August that reader called this
+  > decision a defect in four places** (R-370), and one session's work went into correcting the record
+  > rather than into the product.
 - **Backup scoping:** hot data (LXC rootfs) rides the guest `vzdump` → tiers + PBS. Bulk data
   on external mount points is **excluded** from the guest vzdump (per-mount `backup` flag) and
   gets its own per-volume policy (file-level to a tier, slower cadence — or explicitly *not*

documentation/architecture/02-controller-module-map.md

@@ -1,5 +1,20 @@
 # Felhom Controller Architecture — Part 2: Controller Module Map
 
+> **How to read this document.** Two kinds of statement appear, and where this document marks them it
+> marks them like this — the same wording as `07-backup-architecture.md:11-17`, carried here on
+> 2026-08-22 (R-376) so a reader meets one convention and not eight:
+>
+> - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet.
+> - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation.
+>
+> **Statements in this document are NOT yet all marked.** Marking them wholesale is a large judgement
+> exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376).
+> **An unmarked statement therefore means "not yet classified", never "observed".** That ambiguity is
+> exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat
+> unmarked beside a marked `[FACT]`, and was read as an observation and reported as a defect.
+
+
+
 > **EXECUTED (slice 8C, 2026-06-10 — controller v0.37.0).** This map's target state is now realized:
 > the disk-execution subsystem (`storage/*`, restic, cross-drive, drive-restore, `disk_layout`,
 > `local_infra`, `infra_backup`, `setup/scanner`, `monitor/watchdog`+`pinger`, the storage UI) is
@@ -320,7 +335,7 @@ own; every caller that is not the customer must decide for itself whether the ap
 ### `sync/`
 | File | Class | Reason | Risk |
 |---|---|---|---|
-| `sync/sync.go` | **KEEP** | Catalog git-sync (clone/fetch/reset, copy compose+`.felhom.yml`, never overwrite app.yaml). | clean |
+| `sync/sync.go` | **KEEP** | Catalog git-sync (clone/fetch/reset). **Since controller v0.235.0 it RENDERS `docker-compose.yml` rather than copying it** — verbatim while the catalog still offers the app's pinned version, from the app's stored `applied-compose.yml` once the catalog moves past it. `.felhom.yml` is still copied verbatim always, and `app.yaml` is still never touched. Reasoning: `09-update-architecture.md` §5. | clean |
 
 ### `system/` — split per-function (not per-file)
 | File | Class | Reason | Risk |
@@ -488,3 +503,90 @@ own; every caller that is not the customer must decide for itself whether the ap
 - **S6:** §5(3) self-restore-test → **status-display only**; the agent owns orchestration.
 - **Self-update resolved (03 §11):** `updater.go` → **DELETE(→agent)**, `state.go` →
   DELETE(obsolete), `version.go` KEEP; §6 + §5(2) updated (bulk = `backup=0` mountpoint recipe).
+
+
+---
+
+## The app-definition seam — what happens to a deployed app when its compose file is rewritten
+
+> **STATUS: MEASURED BEHAVIOUR, NOT A RECORDED DESIGN.** This section describes what the system was
+> observed to do on 2026-09-01 (`audits/SPIKE-app-update-2026-09-01.md`), with controls. It is
+> deliberately **not** marked `[DESIGN]`, because the operator has not ruled on whether the behaviour
+> was intended. **Do not read this as an endorsement, and do not spec against it as though it were
+> settled.** The open questions are R-438 and R-441.
+
+Until this was measured, no architecture document said what happens here, and the gap itself is
+R-438. The three facts below are the ones a reader needs before touching any of it.
+
+> **⚠ SECTIONS 1 AND 2 DESCRIBE THE BEHAVIOUR UP TO CONTROLLER v0.234.0. Controller v0.235.0
+> (2026-09-06) CHANGED IT, on an operator ruling.** They are kept because they are the measured
+> account of why it was changed, and because every box below v0.235.0 still behaves this way. **What
+> ships now: `09-update-architecture.md` §5.** In one sentence — an app's VERSION is frozen to what the
+> customer has and only a deliberate Update moves it, while template CORRECTIONS and the self-healing
+> below still arrive on the 15-minute cycle. Nothing was added to the thirteen call sites in §2; they
+> were made safe by removing the reason.
+
+### 1. The catalog syncer rewrites the file under a running app, on a 15-minute cycle
+
+`Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies
+`docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether
+the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only
+exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351`), plus
+one immediate sync at controller start (`sync.go:98`).
+
+**SINCE v0.235.0** this walk still happens and `.felhom.yml` is still copied unconditionally, but the
+compose file goes through `Syncer.renderSource`, which consults a per-app plan supplied by the stack
+manager (`Manager.RenderPlanFor`) through a nil-safe seam. A nil seam is byte-for-byte the behaviour
+described above.
+
+**It restarts nothing.** The post-sync hook is `stackMgr.InjectMissingFields(updated)` and nothing
+else. So from the moment it runs, a deployed app's *definition* and its *running containers* disagree,
+and they stay that way until something else acts.
+
+### 2. Every lifecycle path resolves that disagreement, silently, by upgrading
+
+`StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in
+`docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it
+is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control
+(unchanged file) that did not even recreate the container.
+
+`RestartStack` carries an explicit in-source comment saying this is deliberate — *"so that … any
+template changes (new images, healthchecks) are picked up"*. **That is a fact about the source, not an
+operator ruling**, and it speaks only for the customer-pressed restart.
+
+**Thirteen non-API call sites across nine files** reach `up -d` without anyone pressing anything. The
+full table is §8 of the spike doc; the three that matter most are:
+
+- `bootrecon.Reconciler.Run` → `StartStack` — `bootrecon/bootrecon.go:269`
+- `backup.AppStopGuard.Recover` → `StartStack` — `backup/appstop_marker.go:283`
+- the drive-return gate, `web.Server.restartStacks` → `StartStack` — `web/intermediary.go:222`
+
+**A plain power cut does NOT trigger this.** Docker's `restart: unless-stopped` restores the existing
+containers on the old image, the reconciler finds no orphan, and it logs so
+(`no boot-orphaned apps (nothing to start)`). The unattended upgrade needs the narrower precondition
+*"and the app did not come back"* — which was measured, and does upgrade.
+
+### 3. Nothing takes a copy first, and the tag cannot be put back afterwards
+
+`UpdateStack` runs `compose pull` then `compose up -d --remove-orphans` and does nothing else — no
+dump, no copy, no hold. The R-361 safety dump (`backup/offbox_reconstitute.go:207`) is **database-only**
+and is not on this path at all; an app with no database gets nothing from it even on the paths where it
+does run.
+
+**And restoring the old tag is not a rollback.** Once an app has migrated its data, the old image
+refuses to start — measured on Nextcloud: *"the version of the data (32.0.9.2) is higher than the
+docker image version (31.0.14.1) and downgrading is not supported"*. The data itself survives; only the
+downgrade is blocked. So the only route back is a **data** restore from a copy taken before the
+update.
+
+**And that route fights this seam** (R-441): `stackAdapter.RecreateStackDefinitionFromUnit`
+(`cmd/controller/main.go:2570`) writes the recovery unit's captured compose — with the OLD pin — into
+the live stack dir, and `copyIfChanged` overwrites it again on the next tick. The overwrite is
+measured; that the restore writes to that path is read, not measured.
+
+### What a change here must not break
+
+- The syncer's overwrite is also the **repair** path: a locally-corrupted compose is replaced by the
+  catalog's version within 15 minutes (observed). Any "don't touch deployed apps" rule loses that.
+- `up -d`-on-restart is what injects `app.yaml` env into a running stack. Reverting to
+  `docker compose restart` would silently stop doing that.

documentation/architecture/03-host-agent.md

@@ -1,5 +1,20 @@
 # Architecture Part 3 — The Host Agent
 
+> **How to read this document.** Two kinds of statement appear, and where this document marks them it
+> marks them like this — the same wording as `07-backup-architecture.md:11-17`, carried here on
+> 2026-08-22 (R-376) so a reader meets one convention and not eight:
+>
+> - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet.
+> - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation.
+>
+> **Statements in this document are NOT yet all marked.** Marking them wholesale is a large judgement
+> exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376).
+> **An unmarked statement therefore means "not yet classified", never "observed".** That ambiguity is
+> exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat
+> unmarked beside a marked `[FACT]`, and was read as an observation and reported as a defect.
+
+
+
 > Status: design draft (decision content). To be grounded by Claude Code against
 > `docs/proxmox-platform.md` and `docs/architecture/02-controller-module-map.md`,
 > then placed at `docs/architecture/03-host-agent.md`.

documentation/architecture/04-control-plane-authorization.md

@@ -1,5 +1,20 @@
 # Architecture Part 4 — Control-plane authorization (operator signing)
 
+> **How to read this document.** Two kinds of statement appear, and where this document marks them it
+> marks them like this — the same wording as `07-backup-architecture.md:11-17`, carried here on
+> 2026-08-22 (R-376) so a reader meets one convention and not eight:
+>
+> - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet.
+> - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation.
+>
+> **Statements in this document are NOT yet all marked.** Marking them wholesale is a large judgement
+> exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376).
+> **An unmarked statement therefore means "not yet classified", never "observed".** That ambiguity is
+> exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat
+> unmarked beside a marked `[FACT]`, and was read as an observation and reported as a defect.
+
+
+
 > Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`.
 > To be reviewed by Claude Code against that spike + `03` §4, then placed at
 > `docs/architecture/04-control-plane-authorization.md`.

documentation/architecture/05-hub-architecture.md

@@ -1,5 +1,20 @@
 # Architecture Part 5 — The Hub
 
+> **How to read this document.** Two kinds of statement appear, and where this document marks them it
+> marks them like this — the same wording as `07-backup-architecture.md:11-17`, carried here on
+> 2026-08-22 (R-376) so a reader meets one convention and not eight:
+>
+> - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet.
+> - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation.
+>
+> **Statements in this document are NOT yet all marked.** Marking them wholesale is a large judgement
+> exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376).
+> **An unmarked statement therefore means "not yet classified", never "observed".** That ambiguity is
+> exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat
+> unmarked beside a marked `[FACT]`, and was read as an observation and reported as a defect.
+
+
+
 > Status: design draft (decision content). To be validated by Claude Code against the **actual
 > felhom-hub source** (`felhom.eu` repo, `hub/`) + Parts 01–04, then placed at
 > `docs/architecture/05-hub-architecture.md`.

documentation/architecture/06-offsite-connectivity.md

@@ -1,5 +1,20 @@
 # Architecture Part 6 — Offsite Connectivity (the backup transport)
 
+> **How to read this document.** Two kinds of statement appear, and where this document marks them it
+> marks them like this — the same wording as `07-backup-architecture.md:11-17`, carried here on
+> 2026-08-22 (R-376) so a reader meets one convention and not eight:
+>
+> - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet.
+> - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation.
+>
+> **Statements in this document are NOT yet all marked.** Marking them wholesale is a large judgement
+> exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376).
+> **An unmarked statement therefore means "not yet classified", never "observed".** That ambiguity is
+> exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat
+> unmarked beside a marked `[FACT]`, and was read as an observation and reported as a defect.
+
+
+
 > Status: **design-of-record** (2026-07-03). Records the settled offsite-backup-transport
 > decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
 > Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all

documentation/architecture/07-backup-architecture.md

@@ -98,7 +98,7 @@ with nothing but their dashboard password. No operator, no ticket, no scheduling
 | 1 | „Visszaállítás indítása" | `POST /backup/restore` | destructive — rebuilds the app from its Tier-1 unit |
 | 2 | „Fájlok visszaállítása" | `POST /backup/tier2/restore` | additive, missing-only |
 | 3 | „Visszaállítás a távoli tárolóból" | `POST /backup/offbox/restore` | non-destructive, to a verification copy |
-| 4 | „Helyreállítás az élő adatok közé" | `POST /backup/offbox/place` | additive, missing-only |
+| 4 | „Helyreállítás az élő adatok közé" | `POST /backup/offbox/place` | additive, missing-only  **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. `audits/SPIKE-restic-restore-test-2026-08-31.md`. |
 | 5 | „Teljes visszaállítás (fájlok + adatbázis)" | `POST /backup/offbox/reconstitute` | destructive to files + DB, never deleting |
 | 6 | „Megosztások visszaállítása" + place | `POST /backup/shares/{restore,place}` | additive |
 | 7 | `.fab` import | `POST` → `apiImportStart` | destructive re-import of one app |
@@ -308,17 +308,33 @@ entry wins; else a `:ro` reader is *excluded*; else a writable bind is *mandator
 | **Tier-2** (`mandatory + optional`) | **7** — the 4 above plus audiobookshelf, komga, romm |
 | legacy resolver path (apps with no block) | **0** — no non-block template binds a namespace path |
 
-> ### ⚠️ UNRESOLVED — two counts of the same thing disagree
+> ### ✔ RESOLVED 2026-08-31 (controller v0.229.0) — **A = 7 · B = 45 · C = 1**
 >
-> | source | count |
-> |---|---|
-> | C9-F1 Phase 0, as shipped | **A = 9 / B = 43 / C = 1** — `felhom.eu/REPORT.md:17-24`, restated `backlog/OPEN-ITEMS.md:31` |
-> | INV Part B.1, independent enumeration at catalog `4252121` | **A = 7 / B = 45 / C = 1** |
+> | source | count | verdict |
+> |---|---|---|
+> | C9-F1 Phase 0, as shipped | **A = 9 / B = 43 / C = 1** — `felhom.eu/REPORT.md:17-24` (since overwritten), restated `backlog/OPEN-ITEMS.md:31` | **WRONG by two apps** |
+> | INV Part B.1, independent enumeration at catalog `4252121` | **A = 7 / B = 45 / C = 1** | **CORRECT** |
+> | Measured at catalogue `459766cb16395fd1d1a66282f5cc6da59ead5924`, 2026-08-31 | **A = 7 / B = 45 / C = 1** | adopted |
 >
-> Both use the same definition ("templates whose Tier-2 copy can hold a readable file leg"). The
-> difference is two apps and **neither number is adopted here**. The Phase-0 enumeration is described
-> in prose but the script is not committed, so the two methods cannot be diffed from the repo.
-> **This must be resolved before either figure is used to size anything.**
+> **The method, so it can be re-run rather than re-argued.** A throwaway `main` inside the controller
+> module drove the PRODUCTION rule over all 53 template directories — `stacks.LoadMetadata` (the single
+> validation choke point, so a rejected `backup:` block degrades to legacy exactly as it does live) →
+> `stacks.ParseComposeClassifiableBinds` → `appbackup.ClassifyBinds` → `appbackup.ComputeCaptureSet` at
+> `TierSecondary`, with the legacy branch falling back to `AppDataBindsPresent` + `AppDataDirNames` as
+> `backup.tier2CaptureSet` does. **A = at least one leg survives that pipeline.** 13 templates carry a
+> valid `backup:` block; 40 are legacy and none of them binds a namespace path, so all 40 are B or C.
+>
+> **A (7):** audiobookshelf, calibre-web, immich, komga, nextcloud, paperless-ngx, romm.
+> **C (1):** bentopdf — it declares no `volumes:` key and no `${…_PATH}` bind at all.
+> **B (45):** everything else.
+>
+> **How the earlier disagreement arose, established rather than guessed.** Phase 0's own write-up
+> (controller `CHANGELOG.md`, v0.183.0) says four apps — plex, jellyfin, emby, navidrome — are in B
+> "only because their single bind is a `:ro` media mount, which `ClassifyBinds` correctly excludes". It
+> applied the `:ro` **default** rule. The two apps it therefore missed are **radarr and sonarr**: their
+> `${USERDATA_PATH}/media/*` and `${USERDATA_PATH}/downloads` binds are **writable**, so the `:ro` rule
+> does not reach them, and they are excluded by an **explicit** `class: excluded` entry instead. 9 − 2 =
+> 7 and 43 + 2 = 45, which is exactly the gap. No catalogue file was changed; this is a measurement.
 
 **[FACT] The class-B consequence is real regardless of which count is right.** For an app whose data
 lives entirely in named volumes, the Tier-2 copy holds a full `recovery-unit/` and **no readable
@@ -333,12 +349,124 @@ refuses **before** stopping the app and names the action that works.
 | tier | captured | read back by that tier's restore | gap |
 |---|---|---|---|
 | Tier-1 | unit incl. volume tars + DB dumps | all of it | none |
-| Tier-2 | unit mirror **+** file legs | `hdd/` and `userdata/` **only** (`tier2_restore.go:101-104`) | **the unit mirror is read by nothing** — `RecoveryUnitPath` resolves to `backups/primary/` (`appbackup/paths.go:46-48`) → **R-102** |
-| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay; the unit is **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **no offsite action unpacks the named-volume tars it captures** → **R-107** |
+| Tier-2 | unit mirror **+** file legs | the file restore reads `hdd/` and `userdata/`; **since controller v0.218.0's Tier-3 sibling and now v0.229.0, a SECOND action reads the unit mirror itself** (`RestoreTier2Unit` → `RestoreFromRecoveryUnitAt`, `tier2_restore.go`) | **CLOSED — R-102** |
+| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay **+ the named-volume tars, replayed from the scratch unit** (`offbox_reconstitute.go` `volReplay`, controller **v0.218.0**); the unit itself is still **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **CLOSED — R-107** |
 
-**[FACT]** Tier-2's gap is the sharper one because of *when* it bites: Tier-2 exists for the case
+**[FACT] 2026-08-22 — the Tier-3 row above was corrected; the Tier-2 row was NOT.** Until controller
+**v0.218.0** this table said *"no offsite action unpacks the named-volume tars it captures"*, and that
+was true from the day Tier-3 shipped until 2026-08-21. **R-107 closed in v0.218.0**: `volReplay`
+(`offbox_reconstitute.go`) replays the scratch unit's `volume-dumps/` into the live named volumes,
+proven live on `demo-hp`. The old sentence is kept here, in the past tense, because a correction that
+erases what was believed leaves the next reader no way to tell a fixed gap from one that was never
+noticed.
+
+**[FACT] 2026-08-31 — the Tier-2 half is now closed too (R-102, controller v0.229.0), and the old
+sentence is kept here in the past tense for the reason the paragraph above gives.** Until v0.229.0 this
+table said of Tier-2: *"the unit mirror is read by nothing — `RecoveryUnitPath` resolves to
+`backups/primary/` (`appbackup/paths.go:46-48`)"*, and that was true from the day Tier-2 shipped until
+2026-08-31. The mechanism was a hard-coded `primary` segment: every reader of a recovery unit could
+only name a path under it. `appbackup` now also exposes four **unit-directory-relative** primitives,
+`Manager.RestoreFromRecoveryUnitAt(stack, unitDir)` holds the restore body, and `RestoreTier2Unit`
+points it at `<dest>/backups/secondary/<app>/recovery-unit/`. Proven live on `demo-hp` **with the
+primary unit moved aside** — `audits/DRILL-r102-tier2-unit-2026-08-31/`.
+
+**Two things did NOT change, and both are load-bearing.** THE SOURCE MOVED; THE DESTINATION DID NOT —
+data still lands in the live named volumes and the live database container. And the FILE restore's
+reach is unchanged: `CanRestore()` still answers only "are there file legs?", and
+`tier2UnitNotCoveredMsg` is still appended where that restore runs, so a clean file result never reads
+as a clean bill of health for the database.
+
+**[FACT]** Tier-2's gap was the sharper one because of *when* it bit: Tier-2 exists for the case
 where the primary drive is lost — and in exactly that case the primary unit is gone while this
-mirror survives on the second drive, unreachable by any customer action.
+mirror survives on the second drive. Until v0.229.0 it was unreachable by any customer action.
+
+**[FACT] 2026-08-23 — taking the undo copy used to DESTROY the app's own database backup (R-361).**
+`writeSafetyDump` called `DumpOne` into the app's own unit directory and renamed the result to
+`pre-restore-*` afterwards. `DumpOne` writes `<stack>-<dbtype>.sql` — the app's canonical dump, the
+name the replay loop matches exactly — so **every safety dump overwrote the app's real backup and then
+moved it away**, leaving the app with no database backup of its own until the next nightly run. A local
+restore-from-unit in that window finds no `.sql` and tells the customer the app never had a database.
+The comment beside it asserted the rename meant it *"can never overwrite the app's real dump"*; it was
+false as written and stood for four months. **Measured before the fix on `demo-hp` 2026-08-22:**
+`docmost` and `bookstack` each held only `pre-restore-*` files and no canonical dump.
+
+Fixed in controller **v0.221.0**: `DumpOneTo` takes an explicit final path and derives its own `.tmp`
+from it, so neither the destination nor the scratch file can collide with a nightly dump running
+beside it. **Proven the only way it can be** — the canonical dump's sha256, unchanged across a
+restore, on both engines.
+
+**[DESIGN] 2026-08-23 — `db_dumps` lists the app's OWN dumps, not the undo copies.** They are local
+material for a restore that went wrong, not part of the app's recovery set: nothing reads them from
+the manifest, and three per app were being pushed off-site permanently for no recovery value. The
+files are neither deleted nor hidden — their visibility is a separate recorded decision and it stands.
+**One consequence, recorded because it bit within minutes:** a stable `db_dumps` lets
+`CaptureRecoveryUnit`'s already-current early return fire, so anything that must happen on every
+capture — such as bounding the undo copies — has to sit ABOVE that check, not after it.
+
+**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
+Recorded here rather than only in a closed register row, because a decision that survives only inside
+a closed work item is a decision nobody will find.
+
+1. **Replay.** The snapshot's `.sql` is imported into the app's database, with only the DB service up
+   (R-47). A pre-restore copy of the LIVE database was taken first and is on disk (R-43); the restore
+   refuses outright if it could not be taken.
+2. **Rollback (controller v0.220.0, R-379).** If the replay fails, the product re-applies that undo
+   copy itself. **The whole set for this run** — an app with two databases gets two undo files, and
+   restoring only the first would leave the other half-written — matched on **the run's own stamp**,
+   never on the `pre-restore-` prefix, because several runs' copies coexist in the same directory. It
+   runs with the DB service still up and before any restart, so the app never observes the half
+   state, and into a **re-discovered** container: the DB-only start re-creates it, so the id captured
+   at dump time is dead by rollback time (v0.220.1, found by the first live run). The app then starts
+   and the customer is told **both** that the restore failed and that their data is as it was.
+3. **Hold (v0.220.0, operator ruling 2026-08-22).** If the rollback ALSO fails, the app is **held
+   stopped**, not started. A running app on a half-written database lets the customer type into it and
+   makes the damage permanent. The hold is persisted, every start path refuses it with a reason and a
+   route, the app-stop marker is ended so nothing auto-restarts it at the next boot, and the app reads
+   **red** rather than green. An operator clears it with `--clear-restore-hold`, which requires a
+   controller restart.
+
+**Why a rollback and not an engine flag.** Postgres gained `--single-transaction` in the same release
+and that does make its replay all-or-nothing — but **MariaDB's DDL is not transactional**, so a
+partial apply there is unavoidable at the engine. Measured 2026-08-22: the same truncated dump left
+Postgres emptied and crash-looping, and left MariaDB with its user data intact, its schema-version
+table wiped to zero rows, and the app reporting `health=healthy, running=true, restarts=0`. The flag
+is a belt; the rollback is the fix. **R-379, R-380.** Evidence:
+`audits/DRILL-r379-rollback-2026-08-22/`.
+
+**[DESIGN] 2026-08-22 — the restore destination is resolved by the same rule as the capture
+destination.** The drive if the app declares one (`HDD_PATH`), the system data path otherwise —
+`Manager.GetAppDrivePath`, one expression, used by `CaptureRecoveryUnit` and, since controller
+**v0.219.0**, by `ReconstituteFromOffsite` and `PlaceOffsiteRestore` too.
+
+**[FACT] 2026-09-01 — the rule now has a FOURTH consumer and the "one expression" sentence is
+true (R-414, controller v0.232.0).** `offboxRestoreScratchDir` — where a restore is UNPACKED, as
+distinct from where it LANDS — never consulted `systemDataPath`, so on a box with no registered
+storage path it refused, and the nightly proof could not run at all. **It was not excluded on
+purpose:** R-356's own commit (`08eb1a6`) records in its tests that *"the prepared scratch still
+resolves to the registered storage path … only the DESTINATION moves"* — it was out of scope, and
+every fixture assumed a registered path exists. The one comment about a `systemDataPath` fallback
+belonged to `PlaceOffsiteRestore`, concerned bulk **userdata**, and R-356 deliberately overruled it.
+
+**But the fallback is SCOPED, and the scoping is the point.** The two callers ask different
+questions, and one predicate answering both is R-356's own defect: a **unit-only** restore (the
+R-87 proof) may fall back to the system data path, because §7 records as `[FACT]` that a driveless
+app's unit already lives there indefinitely and that the same-device placement is *"intended, not
+a defect"*; a **full** restore keeps the R-252 refusal, because it pulls the app's bulk userdata
+onto what §2.2 makes a **state-only** tier.
+
+The refusal that protects a drive app from being restored onto the wrong disk (R-253, R-351) applies
+to apps that **have a drive to get wrong**. It used to be reached by testing `HDD_PATH == ""`, which
+also answered "is this app installed?" — one predicate for two questions. Measured in the catalogue at
+`459766cb1639`: **53 templates, 13 declare `needs_hdd: true`, 40 declare `false`**, so for 40 apps that
+test was permanently true and the off-site restore refused them forever, while they were running,
+with a message telling the customer to reinstall them "in the same place" — a place those apps never
+offer. **An app with no drive is not misconfigured** (§8 of `01-topology-and-trust.md` carries the
+`[DESIGN]` marker); it is the majority case.
+
+Since v0.219.0 the two questions are asked separately: *installed?* of `ListDeployedStacks()`, failing
+CLOSED when there is no provider to ask; *where?* of `GetAppDrivePath`. A third refusal, with its own
+sentence, covers installed-but-no-resolvable-data-root. **R-356**; reasoning also recorded in
+`felhom-controller/CONTEXT.md`.
 
 ---
 
@@ -521,14 +649,28 @@ anywhere under the backup namespace. Full record: `audits/D5-drive-alone-restore
 
 **[FACT]** Two instances, both current:
 
-- **Tier-2 vs primary-drive loss.** Tier-2's stated purpose is surviving the loss of the primary
-  drive. In that failure the primary recovery unit is gone; the surviving mirror on the second drive
-  is `backups/secondary/<app>/recovery-unit/`, which **no code path reads** (§6.3). For the 45-or-43
-  class-B apps the restore is a guaranteed no-op in exactly its designed scenario. → **R-102**
-- **Tier-3 vs guest loss.** Tier-3 holds the volume tars and the DB dump. Reconstitution requires
-  the app to be deployed and skips the unit; the tars are unpacked only by the Tier-1 path, which
-  requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest.
-  → **R-107**
+- **~~Tier-2 vs primary-drive loss~~ — CLOSED 2026-08-31, controller v0.229.0 (R-102).** Tier-2's
+  stated purpose is surviving the loss of the primary drive. It was true until v0.229.0 that in that
+  failure the primary recovery unit is gone while the surviving mirror on the second drive —
+  `backups/secondary/<app>/recovery-unit/` — was read by **no code path** (§6.3), so for the **45**
+  class-B apps (§6.2, count settled the same day) the restore was a guaranteed no-op in exactly its
+  designed scenario.
+  **Tier-2 can now meet its prerequisite in the failure it exists for**, and that is stated plainly
+  because it is the whole point: the restore was run on `demo-hp` **with the primary unit moved aside**
+  and returned 3 volumes of 3 and 1 database of 1 in 28.65 s, byte-for-byte, with the app then reading
+  its own row over TCP with its own credential — and again with the guest's `app.yaml` also moved
+  aside, `secrets recovered=2/2` from the mirrored unit. Evidence:
+  `audits/DRILL-r102-tier2-unit-2026-08-31/`.
+  **What the drill did NOT cover, stated per §8 below:** the full drive-loss journey — a genuinely
+  absent or replaced physical drive — was not run. Only the recovery-unit half was.
+- **Tier-3 vs guest loss — HALF of this closed.** Tier-3 holds the volume tars and the DB dump. It
+  was true until controller v0.218.0 that the tars were unpacked only by the Tier-1 path; since
+  v0.218.0 the reconstitution replays them itself (`volReplay`) → **R-107 CLOSED 2026-08-22**. What
+  is **still** true, and is the part that was never R-107: reconstitution requires the app to be
+  **deployed** and skips the unit, so offsite alone cannot rebuild an app onto a fresh guest. That
+  requirement is a deliberate decision (R-253) — the restore does not choose a customer's drive for
+  them — and since **v0.219.0** it means only what it says: an app that is not deployed. It no longer
+  catches the 40 driveless apps → **R-356**.
 
 ### 7.3 ~~What D5 would change — and why it was blocked~~ — **D5 SHIPPED 2026-07-30 (v0.188.0)**
 
@@ -717,6 +859,24 @@ crosses the line — **R-158**.
 
 ## 8. The failure → recovery matrix
 
+> ### THE ARC IS CLOSED FOR BETA — read this before the blanks below (2026-09-01)
+>
+> **Closed at controller v0.232.0 / hub v0.111.1.** Everything a **customer** does for themselves is
+> finished and proven live — rows **1, 2, 3, 3b, 3c, 6, 7, 14**. Everything only an **operator** does
+> is **deliberately deferred until after beta** — rows **4, 8, 9, 10, 11 (+11b), 12**, each carrying
+> the marker **`[BETA-DEFERRED]`** in its status cell so the set greps as a group.
+>
+> **The blanks in the RTO column below are now blank ON PURPOSE, and that is the whole difference.**
+> Eleven rows carry a blank RTO — 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15 — and only six of those are
+> deferred work. **Row 5** has no recovery to time (a derived copy), **row 13** has no route by
+> design, **row 14** is proven and merely never stopwatched, **row 11b** is a note not a row, and
+> **row 15 is an open DEFECT (R-104) that this line does NOT cover.**
+>
+> **NO STATUS MOVED ON THE DAY THIS WAS WRITTEN, because nothing was proven that day.** A stopping
+> line that promotes a row is a stopping line that lies. Full text, and the conditions that reopen
+> this: `documentation/backlog/OPEN-ITEMS.md`, *"DECIDED — the backup and restore arc is CLOSED FOR
+> BETA"*.
+
 **This is the core artifact.** One row per failure. It is authoritative for recovery routes;
 `00-capability-map.md` stays authoritative for per-capability status.
 
@@ -734,31 +894,71 @@ crosses the line — **R-158**.
 |---|---|---|---|---|---|---|---|---|
 | 1 | **Customer deletes files** | everything else | „Fájlok visszaállítása" — Tier-2 additive merge | **customer** | **39 s** (6 files) | 24 h | **PROVEN** | CAMPAIGN-9 A1 — byte-identical returns, both non-destruction promises kept, bytes re-served by paperless's own API |
 | 2 | **An app's data directory is destroyed** | the guest, the other tiers | same route | **customer** | **46 s** (43 files) | 24 h | **PROVEN** | CAMPAIGN-9 A3 — loss verified by a positive observable (doc download 200 → 404), then 16/16 documents usable |
-| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar |
+| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar. **2026-08-30 — this row KEEPS its PROVEN status and the reason is worth stating: R-353 was a defect in the MESSAGE, not in the mechanism.** The restore really did return what the unit held, every time; what it could not do was say so, because the count was discarded one call deep. Controller v0.226.0 fixed the sentence and changed nothing about the recovery path. A status that measures whether data comes back must not move because a status line was wrong |
 | 3c | *same, with the GUEST GONE (secrets unavailable)* | the drive | Tier-1 unit restore — **the unit carries the portable secrets** | **customer** | **27.6 s** | 24 h | **PROVEN** | D5, §7.4. Before v0.188.0 this row was **NONE**: the data-key gate refused and the customer's own copy of their own data was not a recovery |
-| 3b | *same, for a class-B app via Tier-2* | — | **no route** — Tier-2 never reads the unit mirror | — | | | **NONE** | §6.3; **R-102** |
-| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB; **the volume tars in either copy are unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102**, **R-107** |
-| 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go:359-393` |
+| 3b | *same, for a class-B app via Tier-2* | the Tier-2 copy on the second drive | „Teljes visszaállítás a másolatból" — **Tier-2 unit restore** (`POST /backup/tier2/unit-restore` → `RestoreTier2Unit` → `RestoreFromRecoveryUnitAt`), controller **v0.229.0** | **customer** | **28.65 s** (3 volumes, 1 database, 114.5 MB unit) | 24 h | **PROVEN** (2026-08-31) | `audits/DRILL-r102-tier2-unit-2026-08-31/`. docmost — a class-B app whose Tier-2 run reports **0 leg(s)** — restored **with the primary unit moved aside**: 3 volumes of 3 and 1 database of 1, from `/mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit`. **The observable is the DATA:** an accented Hungarian filename returned byte-for-byte (verified as hex, R-364) and the app read its own row **over TCP with its own credential**; the post-backup discriminator was **gone**, so the tar was genuinely replayed. Repeated with the guest's `app.yaml` also aside → `secrets recovered=2/2` from the mirrored unit. **R-102 CLOSED** |
+| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 of 53 apps) **and, since controller v0.229.0, for the unit mirror — its volume tars and DB dump, i.e. the whole of what the other 45 own**; Tier-3 reconstitute for files + DB **+ the named-volume tars since v0.218.0** (`volReplay`) | **customer** (all) | | 24 h | **PARTIAL** **`[BETA-DEFERRED]`** | §7.2. **Both unreachability gaps are now closed — R-107 (v0.218.0) and R-102 (v0.229.0).** This row stays **PARTIAL** deliberately: what is proven is the ROUTE (row 3b, live, primary unit absent), not the JOURNEY. **No drive has ever actually died or been replaced under this recovery** — the drill removed a unit directory, not a disk, so drive re-attachment by `durable_id`, the agent's enrolment of a replacement, and a Tier-2 copy read from a drive that is the ONLY surviving one are all still unexercised. Promoting this row to PROVEN needs that journey, not another unit restore |
+| 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go`. **The derived-copy rule is UNCHANGED by R-403 (controller v0.230.0) and the single exception is stated in §8.2 below — read it before "fixing" a skip you find in the code** |
 | 6 | **Guest lost or corrupted** | the host, both whole-guest tiers, the data drives (they are host binds) | `pct restore` from `local:` or `felhom-pbs:` | **operator** (SSH) | **84–112 s** local · **1101 s** PBS (both = restore-test into a scratch guest, boot + verify + teardown) | 24 h local · 7 d offsite | **PROVEN** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C (exact mount parity, `unprivileged: 1` preserved); LIVE restore-tests on both boxes this session |
 | 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
-| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
-| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) |
-| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) |
-| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
-| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
-| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D |
+| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
+| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) |
+| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`)  **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`.  **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
+| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
+| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
+| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D |
 | 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) |
 | 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
 | 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
 
+### 8.2 The ONE exception to row 5's derived-copy rule (R-403, controller v0.230.0)
+
+**[DESIGN] Row 5 stands: the secondary IS a derived copy and it IS rebuilt on the next run.** Nothing
+below weakens that, and a future reader who finds `RunTier2` skipping a leg and does not find this
+section will "fix" it back — which is why it is here and not only in a register row.
+
+**[FACT] What was measured, on the shipped v0.229.0, on `demo-hp` 2026-08-31.** An app's Tier-2 copy
+went from **120 082 104 B** (4 database dumps + 3 named-volume tars) to **7 036 B** (none of either)
+in one nightly run, and the run recorded itself a success. The mechanism was three individually
+correct lines: `RunTier2` guarded the unit leg with `os.Stat` alone — *does the folder exist* —
+`rsyncMirror` is `rsync -a --delete`, and nothing between them compared source to destination. **An
+empty recovery unit is a folder that exists.** Evidence:
+`audits/DRILL-r403-tier2-delete-2026-08-31/`.
+
+**Why it bites harder since 2026-08-30:** R-102 made that mirror a LIVE recovery route (§6.3, §8 row
+3b). Deleting it used to cost a copy nobody could open; it now costs the route itself.
+
+**[DESIGN] The exception, stated exactly.** The unit leg — and ONLY the unit leg — is skipped when the
+SOURCE unit carries no data and the DESTINATION unit does. Everything else is unchanged:
+
+| source unit | destination unit | behaviour |
+|---|---|---|
+| complete | complete | mirror, with `--delete`, as before |
+| complete | hollow or absent | mirror (the normal first copy) |
+| hollow | hollow | mirror — both sides agree, nothing is at risk |
+| **hollow** | **complete** | **skip the unit leg, preserve the destination, warn, record for the surface** |
+
+**"Hollow" is a MANIFEST question, never a size question** — the manifest lists no database dump and
+no volume tar; absent or unparseable counts as hollow, fail closed. A unit with a fat compose capture
+and no dumps is the dangerous shape; a 360-byte unit belonging to a tiny app is healthy.
+
+**The data legs are NOT guarded and must not be.** A classified app's copy legitimately shrinks as
+`export` drops out of its class set (`tier2.go` header), and fencing that would be calling this row's
+own decision a defect.
+
+**[DESIGN] And the CAUSE is closed at the other end.** The hollow primary was written by the 5-minute
+capture job **two seconds** after a Tier-2 unit restore. Since v0.230.0 `RestoreTier2Unit` refills an
+absent or hollow primary unit from the mirror **inside the call, before returning**, so no capture can
+observe the hollow state. **The capture itself is deliberately not guarded:** a capture that describes
+an empty drive as empty is correct, and guarding it would make the manifest lie.
+
 ### 8.1 The blank cells, listed explicitly
 
 Per the rule that a blank is a finding, here they are:
 
 | row | blank | why |
 |---|---|---|
-| 3b | RTO, RPO | no route exists to time |
-| 4 | RTO | no drive-loss recovery has ever been timed |
+| 4 | RTO | no drive-loss recovery has ever been timed. **The Tier-2 unit restore inside it now is** — 28.65 s, row 3b, 2026-08-31 — but that is the route, not the journey: no drive has been removed or replaced under a recovery |
 | 5 | RTO | never timed; the rebuild is a normal Tier-2 run |
 | 6 | — | RTO present, but it is a **restore into a scratch guest on the same host**; a restore *to a different host* has never been timed |
 | 8 | RTO | a host has never been rebuilt as itself (INV Part D1) |
@@ -899,12 +1099,21 @@ does **not** hold as written. → **R-108**
 | **R-104** | An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach, reported as *„ismeretlen okból"* | the offsite tier stays dead until a human unlocks. Was C9-F3 |
 | **R-105** | Three hub-held DR records are empty on the whole live fleet: `hosts.dr_record_json`, `host_escrow.directive_json`, `dr_recipe.host_half.drives` | the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field |
 | **R-106** | `dr_recipe.host_half.pbs.namespace` records `"root"` on every box | the recorded restore coordinate is wrong; real namespaces are per-customer |
-| **R-107** | No offsite action unpacks the named-volume tars Tier-3 captures on every run | offsite alone cannot rebuild a named-volume app (§7.2) |
+| **R-107** | ~~No offsite action unpacks the named-volume tars Tier-3 captures on every run~~ — **CLOSED, controller v0.218.0, 2026-08-22** (`volReplay`, proven live on `demo-hp`). True from the day Tier-3 shipped until 2026-08-21. | was: offsite alone cannot rebuild a named-volume app (§7.2). Now: the tars replay; what remains is that the app must be **deployed** (R-253), which is not R-107 |
+| **R-356** | The off-site restore resolved its destination with the raw `HDD_PATH` and read an empty answer as "not installed" — **CLOSED, controller v0.219.0, 2026-08-22** | 40 of 53 apps were refused permanently while running (§6.3 `[DESIGN]`) |
 | ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
 | **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
-| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
+| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo, from **two** call sites (`offbox.go:1388` retention and `offbox.go:1759` over-quota) | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10). **SPIKE 2026-09-01:** prevention needs a transport change (restic 0.14.0 does speak `rest:` — measured; append-only is a rest-server flag, not a restic one), because the sub-account API cannot express write-without-delete. **Detection is nearly free and is recommended first** — `snapshot_count` already reaches the hub and the hub appends reports, so the comparison needs no box change. `audits/SPIKE-r95-offsite-delete-2026-09-01.md`. **RE-SCOPED 2026-09-01:** the box can delete the LIVE repository but **cannot write to the daily snapshots of it** (measured, both boxes) — so the exposure is at most one day's data plus an operator-driven per-file recovery, not open-ended loss. Detection shipped hub v0.111.0 (R-431). **DRILL 2026-09-01: the re-scope's recovery clause is WITHDRAWN — no snapshot is reachable from a sub-account by any name (R-433, 777,600 names, controlled), so the exposure is NOT bounded by an operator-driven per-file recovery. R-436 is the new cheap lead: the provider already offers `rclone serve restic --stdio` server-side and restic 0.14.0 speaks `rclone:` (measured) — but the client supplies the server command line, so ask the vendor whether `--append-only` is pinned BEFORE building anything.** |
 | ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
-| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
+| ~~R-353~~ | ~~A local unit restore reported a bare completion whether it returned an entire dataset or nothing~~ | **CLOSED 2026-08-30, controller v0.226.0.** The volume count already existed (`restoreDockerVolumesFrom`) and was discarded by a one-line wrapper, so the surface was structurally unable to say what came back. `RestoreFromRecoveryUnit` now returns `UnitRestoreResult` and the sentence has three cases, keyed on replayed-vs-**listed** — zero-replayed has two causes that are opposite news. Every sentence is a claim about the BACKUP, never the app: this path has no `SafetyDump` discriminator, and §6.3 is why that is not pedantry. Proven live on `demo-hp` |
+| ~~R-357~~ | ~~The destructive reconstitute had no free-space gate; all three that existed guarded non-destructive paths~~ | **CLOSED 2026-08-30, controller v0.226.0.** The gate sits before `writeSafetyDump` and `StopStack`, so a refusal costs no outage — the test asserts the StopStack call count, not the sentence. No headroom multiplier (a measured tree, not a predicted download); fail-closed when either probe reads ≤ 0, which was a real fail-open hole. **Not live-validated** — filling a filesystem is a drill step |
+| ~~R-358~~ | ~~`OffboxFullScratchReady` asked "non-empty directory", which is exactly what a failed restic run leaves~~ | **CLOSED 2026-08-30, controller v0.226.0.** A completion marker written only after restic returns nil, stale ones cleared before it starts, both orders pinned by an AST test. Both handlers refuse server-side — a hidden button is not a guard. Proven live on `demo-hp`. **See R-396: the bad state needed only a SUCCESSFUL safe restore, not a failed one** |
+| ~~R-360~~ | ~~The verification-copy delete gated on the concurrency flag, which a verification restore never holds~~ | **CLOSED 2026-08-30, controller v0.226.0.** `restoreOpBlocked()`, matching the five sibling handlers. The doc comment that claimed it already did this is corrected in place — that sentence is why nobody looked. Proven live on `demo-hp` in the exact flag state that produced it |
+| ~~R-396~~ | ~~A unit-only verification restore unlocked the destructive full restore~~ | **CLOSED 2026-08-30, controller v0.226.0** (by R-358's marker). Found while answering R-358's open question. Both restore modes write the SAME scratch directory, and one boolean (`ScratchReady`) drove three different intents — "is there a scratch", "may we place", "may we destructively restore". The safest action on the page unlocked the most dangerous one |
+| ~~R-359~~ | ~~The off-site restic store is never verified by anything, ever~~ | **CLOSED 2026-08-30, controller v0.227.0/v0.227.1.** A daily `offsite-integrity` job on **due-ness, not a weekday**; it takes the single-writer flag and SKIPS rather than waits (`resticStep` escalates to `unlock --remove-all` and is only safe while that flag is held). Three outcomes — skipped / unreachable / failed — because 'I could not look' is not 'I looked and it is broken'. **⚠ The depth that ships ON does NOT catch silent corruption:** measured, a pack corrupted without a size change returned `no errors were found`, exit 0; only `--read-data*` caught it. Choosing the depth is **R-399** |
+| ~~R-397~~ | ~~`NotifyIntegrityOK`/`NotifyIntegrityFailed` had no caller and the product advertised a weekly check that did not exist~~ | **CLOSED 2026-08-30, controller v0.227.0.** Sixth built-but-never-wired instance: hub allowlist, Hungarian text, settings checkbox and debug button all existed; only the caller did not. `ok` is severity `info` and mails nobody by design |
+| ~~R-399~~ | ~~The check reads the catalogue and never the data~~ | **CLOSED 2026-08-31, controller v0.228.0.** `monitoring.integrity.read_data_subset` now defaults to **`100%`**, so the weekly check downloads and re-hashes every stored byte. **The fact that made it necessary, and the sentence that should stop anyone turning it back down to save four seconds: the structure check PASSED a size-preserving pack corruption.** Measured on `demo-hp` 2026-08-30 — plain `restic check` reported `no errors were found` and exited 0 over a pack damaged without a size change; every read-data form caught it. Cost on that 134 MB store: 35.0 s structure vs 39.2 s at 100%. `off` (any case) returns a box to structure depth; an empty value means *not configured*, therefore the default; a malformed value WARNs and falls back to the DEFAULT, never to structure. A completed check over 5 minutes logs an operator WARN naming R-401 — **one data point, on one 134 MB store, so no rotation schedule, size threshold or bandwidth budget was invented from it.** Proven live at both depths 2026-08-31 with the restic argv observed from the guest |
+| ~~**R-87**~~ **— SHIPPED 2026-08-31 (controller v0.231.0), RE-SCOPED. AND IT IS STILL NOT R-359** | ~~The restic tier is never restore-TESTED~~ **The box now PROVES its own off-site copy still HOLDS something.** **THE DISTINCTION, STATED ONCE AND PLAINLY BECAUSE THE GREEN TICK INVITES THE OTHER READING: this proves the snapshot CONTAINS a recoverable unit; it does NOT prove a restore puts data back into a running app.** The proof restores to a throwaway folder, judges it against the unit's own captured compose, and deletes it — it never touches a live app. Putting data back is drill work. **§8 matrix row 4 is deliberately NOT moved.** Nightly at 05:30, one app, due-ness per SNAPSHOT (R-86's model). Evidence `tests/r87-offsite-proof-2026-08-31/`; the reasoning is `audits/SPIKE-restic-restore-test-2026-08-31.md`. | **SPIKE VERDICT, `audits/SPIKE-restic-restore-test-2026-08-31.md`:** build the NARROW version, not the row as written. **Measured:** a scratch restore of all 8 apps costs **25 s / ≤213 MB scratch**, LESS than the 40.3 s weekly check beside it; but restic 0.14.0's `--verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving corruption passed clean, red-proofed), `restic ls --json` carries no content hash, and the unit manifest hashes **4 918 B of a 213 231 242 B unit** — so **no reference for "correct" exists** (R-409). **Of the five drill-found restore defects R-353/354/356/358/403, an unattended scratch-restore would have caught ONE (R-356).** The value is elsewhere and the weekly check structurally cannot reach it: `check` proves the stored bytes are the stored bytes, never that we stored the RIGHT thing — a hollow unit backs up, checks and restores cleanly and recovers nothing (R-403, measured 2026-08-31). **Proposed re-scope, Viktor's call:** *prove the off-site snapshot still CONTAINS a recoverable unit* — one app a night, restored to scratch, checked against its own `manifest.json` via the existing `unitCarriesData`. Must use `--no-lock` and skip `unlockStale` (R-95's constraint is otherwise violated — R-407/R-408 record what the path writes today) and must take `acquireRunning`, which `RestoreOffboxScratch` does not. | matrix row 4's route has no unattended proof. **Stated explicitly because the two rows sit next to each other and are easy to conflate: a `check` proves the STORE IS READABLE; a restore-test proves DATA COMES BACK OUT.** v0.227.0 did the first and nothing else. This row is untouched by it |
 
 ### 10.3 Divergences that are documented elsewhere and are not re-opened here
 
@@ -938,6 +1147,15 @@ and PBS (Cloud server ep0 + a Cloud Volume) are both Hetzner. Whether they share
 login and payment method** is unverified (INV Unknown U-1) — the question was deliberately not
 answered by calling the provider API with the production token. Accept explicitly, or mitigate.
 
+> **The fence was reached again, and held, 2026-09-01 (SPIKE R-95).** The R-95 spike needed one
+> provider-side fact — **do daily snapshots exist on the Storage Box, and how many** — because the
+> register calls that mitigation ARMED and the whole ranking of R-95's options turns on it. It is
+> readable from the box: measured on BOTH machines over their own SFTP credential, with positive
+> and negative controls, **no `.snapshots` is visible to either sub-account**, and the account is
+> jailed at `/`. The register's own confirming field, `size_snapshots`, is an API field. **So the
+> spike stopped here rather than answering it by acting**, and left it as R-429 — ten minutes in
+> the Storage Box panel, and it re-ranks R-95. `audits/SPIKE-r95-offsite-delete-2026-09-01.md` §Q1.
+
 **E. `local` vzdump shares a physical device with the guest it backs up.** LIVE on both hosts:
 `/var/lib/vz` (the archive target) and `local-lvm` (the guest's rootfs and both `backup=1`
 mountpoints) are both on `/dev/sda3` → VG `pve`. The tier therefore protects against **corruption

documentation/architecture/08-alarm-ladder.md

@@ -0,0 +1,293 @@
+# 08 — The app-down alarm ladder
+
+**Written 2026-08-23, with controller v0.222.0 (R-384).**
+
+**The absence is the finding.** Until this file existed, no document owned the question *"when does a
+customer's app being broken raise an alarm?"* The rules were spread across four packages as comments,
+each locally correct, and the ordering between them was legible only by reading
+`aggregateState` top to bottom. That is exactly how R-384 survived: every individual rule was right,
+and the composition was wrong. Three separate defects in this ladder (R-51, C9-F2, R-384) were each
+found on live hardware rather than by review, and each is a case where a reader could not see the
+whole ladder at once.
+
+Everything below is **[DESIGN]** — deliberate, with the reason recorded — unless marked otherwise.
+
+---
+
+## 1. The two questions, and their order
+
+Two different questions get asked about a multi-container app, and **the order between them is
+load-bearing**:
+
+1. **Is a SUPERVISED member of this app dead?** — a container Docker's restart policy says should be
+   running, that is not.
+2. **Is a RUNNING member failing its healthcheck?**
+
+**Question 1 is asked FIRST.** [DESIGN, R-384, v0.222.0]
+
+Until v0.222.0 it was asked second, and the consequence was not subtle: a two-container app whose
+database exits goes `unhealthy` seconds later *because it cannot reach that database*. So the symptom
+the dead database causes was the thing that suppressed the alarm for it. Measured on `demo-hp`
+2026-08-22 — `bookstack-db` stopped at 21:27:01 and the watcher reported `0 currently down`
+throughout.
+
+**"Some members are up" means any member NOT in the down bucket** — `running`, `unhealthy`,
+`starting` or `restarting`. [DESIGN, R-384] The earlier guard was `running > 0`, counting only
+`StateRunning`, which made the supervised test unreachable in precisely the case it was written for:
+an unhealthy survivor beside a dead database counted as nothing being up.
+
+---
+
+## 2. Where each decision is made
+
+| Decision | Where | Notes |
+|---|---|---|
+| container → stack aggregate state | `internal/stacks/manager.go` `aggregateState` | the ladder in §3 |
+| is a down member supervised? | `internal/stacks/manager.go` `supervisedPolicy` | `no`/`on-failure` benign; everything else, **including unknown**, supervised |
+| which states mean "down" | `internal/stacks/manager.go` `IsDownState` | `{stopped, exited, degraded}` |
+| stack state → "this app is down" | `cmd/controller/main.go` `classifyRunStates` | **the single derivation point**; all three suppressions live here |
+| sustained restarting → down | `internal/stacks/manager.go` `CrashLooping` | 5-minute threshold |
+| quiesce suppression | `internal/quiesce/suppress.go` | cycle-keyed, 180 s grace |
+| boot repair | `internal/bootrecon/bootrecon.go` | consumes `IsDownState` |
+
+---
+
+## 3. The aggregation ladder, in order
+
+`aggregateState(containers, policyOf)` — **priority: degraded > unhealthy/starting > restarting >
+all-running > stopped.**
+
+1. no containers → `not_deployed`
+2. **any DOWN member is supervised, and any member is up → `degraded`** ← R-384 put this first
+3. any `unhealthy` → `unhealthy`
+4. any `starting` → `starting`
+5. any `restarting` → `restarting`
+6. all running → `running`
+7. all down → `stopped`
+8. mix, every down member benign → `running`
+
+Step 2's `policyOf` is consulted **only** for the down members, and only when something is up. `nil`
+is allowed; every down member then reads as supervised.
+
+---
+
+## 4. Which states alarm, and which deliberately do not
+
+`IsDownState` = `{stopped, exited, degraded}`.
+
+| State | Down? | Why |
+|---|---|---|
+| `stopped`, `exited` | **yes** | not running, will not recover alone |
+| `degraded` | **yes** | [DESIGN, R-51] a dead supervised member is as unreachable as a single app that exited — immich-server sat Exited 18 h with the app 100 % dead and no alert |
+| `unhealthy` | **NO** | [DESIGN] a running container whose healthcheck is failing. Folding it in reintroduces the flapping fix-3 was written to stop. **R-384 did not change this** — it asks a prior question instead |
+| `restarting` | **NO**, until sustained | [DESIGN, C9-F2] `restarting` is on the normal deploy path, so folding it in would alarm fleet-wide on every update. Becomes down after **5 min** (`crashLoopAfter`) |
+| `starting`, `deploying` | no | mid-start |
+| `paused` | no | a deliberate user action |
+| `unknown` | no | [DESIGN] fail-OPEN — never manufacture a dead-app alert from an inconclusive read |
+
+**Note the two fail directions are deliberately opposite.** `IsDownState` fails OPEN on `unknown`
+(ambiguous *state* → do not alarm). `supervisedPolicy` fails CLOSED on unknown (we already KNOW a
+member is dead; only the excuse is missing). Both are recorded at their sites.
+
+---
+
+## 5. The three suppressions, all at `classifyRunStates`
+
+| Suppression | Rule | Expires? |
+|---|---|---|
+| **deliberate user stop** | `StateStopped` is not down **unless** the quiesce loop reports it failed to restart that stack | n/a — lifted by `failedRestart` |
+| **quiesce cycle** | a stack this backup cycle stopped is exempt | **yes**, 180 s after unquiescing |
+| **boot grace** | no evaluation for 90 s after controller start | **yes** |
+
+**None of them latch.** [DESIGN, R-97b + R-88 Scenario D] A permanent suppression trades a loud false
+alarm for a silent real one, which is the same error as an over-eager alarm, in the opposite
+direction. Every window expires; the cost is a bounded DELAY in reporting a real failure, never its
+loss.
+
+The quiesce suppression is **cycle-keyed, not state-keyed** — an app caught mid-restart is
+`starting`/`unhealthy`, not `stopped`, so no state test can see it. It is therefore **state-blind**,
+which is why R-384 moving a stack from `unhealthy` to `degraded` cannot weaken it.
+
+---
+
+## 6. The alarm itself
+
+Edge-triggered: `app_start_failed`, one event per transition into down, **not** per scan. Verified
+live 2026-08-23 — one event across 22 scans.
+
+- **Operator/hub event + dashboard banner.** `app_start_failed` is **not** in
+  `settings.DefaultEnabledEvents`, so by default it does **not** e-mail the customer.
+- **F-OBS heartbeat**, every 20 scans (~10 min), at `[INFO]`:
+  `[deadapp] check alive: N scans since boot, M deployed app(s) evaluated, K currently down`.
+  This line exists because an absent alarm and a stopped detector look identical in a log.
+
+### 6.1 The severity contract [DESIGN, R-329 — CLOSED controller v0.223.0 / hub v0.107.0]
+
+**The vocabulary is the HUB's and it is exact: `{info, warning, error, critical}`.** Anything else is
+**coerced to `info` at ingest**, and `info` is dropped by `severityNotifies` before *both* delivery
+legs. So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
+and is e-mailed to **nobody**.
+
+**This shipped twice.** `DiskAlertKind.Severity` emitted `"warn"` until controller v0.215.0;
+`app_start_failed` emitted it until v0.223.0. Measured on the live hub DB 2026-08-23: **91
+`app_start_failed` events stored all-time, ZERO `notification_log` rows before that day** — not one,
+on any channel.
+
+Three things now hold it:
+
+1. **The emitter is pinned by an AST walk** over the whole controller
+   (`TestR329_EveryEmittedSeverityIsInTheHubVocabulary`). Not grep — "warn" is a legitimate
+   *healthcheck status* in `internal/monitor` and `internal/selftest`. The six call sites that pass a
+   variable are registered by name with the values each can take, so a new dynamic path fails.
+2. **The hub SAYS SO** when it coerces (hub v0.107.0, R-387): a `WARN` naming the customer, the event
+   type and the rejected value. **The coercion stays** — a rejected event is a *lost* event, and
+   losing an alarm is worse than mis-routing one.
+3. **The dispatcher's `unrecognized severity` branch is kept**, because the hub's own monitor checkers
+   call `ProcessEvent` directly and never pass the ingest handler. For them it is the only guard.
+
+**Who gets it.** `processOperator` consults only `operatorOn`, the address and a 1-hour cooldown —
+**never customer preferences** — so a valid severity always reaches the operator. `processCustomer`
+consults `operatorOnlyEvents` and then the customer's `enabled_events`.
+
+**`app_start_failed` is customer-switchable but OFF by default** [DESIGN, operator ruling 2026-08-23]:
+it is deliberately absent from `DefaultEnabledEvents`, and deliberately **not** in `operatorOnlyEvents`
+— being in that register would make the toggle visible, flickable and structurally unable to deliver.
+
+**`backup_integrity_ok` / `backup_integrity_failed`** [DESIGN, R-359/R-397 — controller v0.227.0]. Both
+existed in `internal/notify` with **no caller** until v0.227.0 wired them; the hub had allowlisted both
+and carried the Hungarian customer text for both the whole time.
+
+| event | severity | reaches | why |
+|---|---|---|---|
+| `backup_integrity_ok` | **`info`** | **NOBODY** | `severityNotifies` drops `info` before both legs, and that is the intended outcome, not an oversight. **A weekly success e-mail is how people stop reading their alerts.** It is still pushed and stored, because the event stream is where "was it checked?" is answered — the dashboard reads it, the inbox does not |
+| `backup_integrity_failed` | `error` | operator always; customer if enabled | the customer's backups may be damaged, which is the loudest fact this tier can produce |
+
+**`backup_integrity_failed` is deliberately in NONE of the three registers**, and all three were checked
+rather than assumed (2026-08-30):
+
+- **not** in `perAppCooldownEvents` — there is ONE store, not one per app. The coarse per-type hourly
+  key is correct here, and adding it would be a fenced act under §6.2 for no gain.
+- **not** in `operatorOnlyEvents` — the operator leg ignores customer preferences anyway, so the
+  operator is always mailed; putting it here would only remove the customer's ability to opt in.
+- **not** in `DefaultEnabledEvents` — customer-switchable, default OFF, the same ruling as
+  `app_start_failed`. The checkbox already exists at `settings_notifications.html:34`.
+
+**And a caveat that belongs in the alarm ladder rather than only in the backup document:** a
+`backup_integrity_ok` at the shipped depth means *the index, the pack inventory and the snapshot graph
+are sound*. It does **not** mean the stored bytes were re-read — measured 2026-08-30, a pack corrupted
+without a size change passes the structure check with `no errors were found`. An `ok` here is a real
+signal about a real class of failure, and it is narrower than the phrase suggests (R-399).
+
+---
+
+## 6.2 The delivery grain — how often, and per what [DESIGN, R-389 — hub v0.108.0]
+
+**"How loud" is a separate decision from "does it alarm", and it is made in one place**: the operator
+cooldown key at `processOperator`. Everything sharing a key is collapsed for **one hour**.
+
+| Family | Grain | Key carries | Why |
+|---|---|---|---|
+| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |
+| backup run (`backup_run_failures`) | per RUN | `…:<run_id>` | a digest already lists every failing app; one per run |
+| tiered backup (`whole_guest_backup_failed`, …) | per TIER | `…:<tier>` | the tiers fail independently and mean different things |
+| everything else, incl. `crossdrive_failed` and `backup_integrity_failed` | per TYPE, per hour | — | coarse **on purpose** |
+
+**The default is COARSE and that is deliberate.** R-97a and R-182 exist so that one full disk produces
+**one** mail listing every affected app rather than one per app. Widening the grain is what makes an
+operator stop reading their alerts, which is the same failure as not sending them.
+
+**`app_start_failed` is the exception because it has no digest.** There is no `apps_down_run`
+summarising a scan the way `backup_run_failures` summarises a run, so per-app is the only grain
+available that does not lose alarms. Until hub v0.108.0 it was keyed per type, and **only the first
+broken app per hour reached the operator** — measured 2026-08-23: `bookstack` sent at 09:27:51,
+`privatebin` suppressed at 09:31:51 under `key=demo-hp:app_start_failed`.
+
+**The mechanism is a named ALLOW-LIST (`perAppCooldownEvents`), not a payload rule**, and the
+distinction is load-bearing rather than stylistic: **`crossdrive_failed` is severity `error`, reaches
+the operator leg, and carries `stack_name`** through `CrossDriveDetails`. A rule of the form "if the
+details carry a stack_name, split per app" would have split it, silently, and undone R-182.
+`cooldownStackSuffix` therefore takes the **event type** as well as the details — an asymmetry with
+its two siblings, and the reason for it is exactly this.
+
+**Fenced act:** adding an entry to `perAppCooldownEvents` for a type whose family has a digest, or
+whose coarse cooldown is deliberate. Reading the register anywhere is fine.
+
+**Measured burst, so the volume is a number and not an impression:** three apps stopped in one scan
+produced **three attempted, three sent, zero suppressed** (2026-08-23). The reference box has 8
+deployed apps, so a total outage is 8 mails. The boot grace (90 s), the quiesce grace (180 s) and the
+per-app edge trigger absorb reboots, backup cycles and persistently-dead apps. **This scales linearly
+with app count and has no ceiling** — the condition that would reopen the question is a box large
+enough that a total outage is unreadable, at which point the answer is a digest with a customer
+message, not a wider cooldown.
+
+---
+
+## 7. The intent test [DESIGN, R-386 — CLOSED controller v0.223.0]
+
+**"The customer stopped this" is asked of the FIELD THAT RECORDS IT, never inferred from the state.**
+
+Until v0.223.0 `classifyRunStates` read `st.State == StateStopped` and assumed every stopped stack was
+deliberate. It is not inferable: `aggregateState` folds `StateExited` into the stopped counter, so an
+all-down stack returns `StateStopped` whatever killed it. Measured on `demo-hp` 2026-08-23:
+`privatebin` stopped out of band, nine dead-app scans over four minutes, **zero events, zero banner
+lines** — while a comment beside the code claimed an out-of-band stop *"still alerts"*.
+
+`DesiredState` records the answer, has **exactly one writer** (the customer's own action), and is
+tri-state:
+
+| Intent | Verdict | Why |
+|---|---|---|
+| `Stopped` | **no alarm** | the customer asked |
+| `Running` | **ALARM** | nobody asked — the R-386 case |
+| absent (`""`) | **no alarm, and SAY SO** | UNKNOWN never means running |
+
+**The absent case keeps the old behaviour deliberately.** Reading it as "nobody asked" would, on the
+first cycle after upgrade, e-mail about every app any owner ever stopped — fleet-wide, from a field
+that predates the intent being asked of it. The backfill cannot help: it seeds `Running` only from an
+observed-**up** reading, so anything stopped at upgrade time stays unknown, which is precisely the
+ambiguous population.
+
+**The gap is BOUNDED, not silent.** Every such suppression sets `AppRunState.IntentUnknown`, and the
+scheduler logs the names at `INFO` on the heartbeat cadence:
+
+```
+[deadapp] N stopped app(s) have NO recorded customer intent, so their dead-app alarm is
+suppressed by the unknown-intent fallback (R-386): <names>. This closes itself as each app is
+started or stopped through the interface.
+```
+
+**A rule without a mechanism is a wish.** Measured on `demo-hp` 2026-08-23: **0 of 8 deployed apps had
+an absent intent** — the population is already empty on an exercised box; it will be larger on one
+upgraded and left alone.
+
+`failedRestart` still lifts a `Stopped` intent, and that ordering is load-bearing: the quiesce loop
+stops stacks by the same path a customer does, so one it stopped and could not restart must alarm
+whatever the intent says. Removing that term re-opens F-CRIT-1.
+
+**Fenced act:** adding a `DesiredState` **writer**. Reading it anywhere is fine. Twelve of
+`StopStack`'s fourteen callers are machines, so recording intent in the primitive would make a nightly
+backup indistinguishable from the customer pressing Stop.
+
+---
+
+## 8. Direction — who a customer should be notified about at all
+
+**[DESIGN — DIRECTION, NOT CURRENT BEHAVIOUR. Dated 2026-08-23, the operator's own framing.
+Nothing in controller v0.223.0 / hub v0.107.0 implements this.]**
+
+> **A customer should be notified only about things they can act on or are responsible for** — the
+> drive they unplugged, the storage they filled. **A failed backup is our incident, not theirs.** The
+> intended shape is that we detect it, we tell them we noticed and are dealing with it, and they are
+> not handed an error they cannot solve. The subscription should feel like being looked after, not
+> like being on call.
+
+Today's settings page is the opposite shape: it exposes one toggle per detector and **grew from 12 to
+15 in this session alone** (one new alarm, plus two compound toggles split into four). That growth is
+the argument, not an aside — a page that grows by one per detector is a page that will keep asking a
+household to make engineering decisions.
+
+`app_start_failed` defaulting **off** is consistent with this direction and reversible either way; it
+was ruled that way on its own merits and does not pre-judge the redesign.
+
+**Filed as a PRODUCT DECISION, not a defect** — see the register. It is the operator's call to take
+separately, and no part of it was implemented here.

documentation/architecture/09-update-architecture.md

@@ -0,0 +1,451 @@
+# 09 — How an app update works, and what it is becoming
+
+> **LIVING DOCUMENT. Every slice of the update arc updates this file in the same session.**
+> Opened 2026-09-02 with slices 1 and 2. Its absence was **R-438**: the update mechanism was chosen
+> deliberately and written down nowhere, which is how a deliberate design gets "fixed" by someone who
+> did not know it was one.
+
+**This file carries the REASONING. The register (`backlog/OPEN-ITEMS.md`) carries the work. The
+source is the truth.** Nothing here is invented: every mechanism claim is cited either to
+`audits/SPIKE-app-update-2026-09-01.md`, which measured it live, or to live source at `file:symbol`.
+
+---
+
+## 1. How an update works today, as measured
+
+### 1.1 The button
+
+`Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose
+commands and nothing else:
+
+```
+compose pull                       →  compose up -d --remove-orphans
+```
+
+**No safety copy. No rollback. No hold. No verification.** Confirmed by reading and across six live
+updates (spike §10 item 6). A pull FAILURE is handled correctly — `UpdateStack` returns after the
+failed pull and never reaches `up -d`, so the running app survives untouched (measured twice, spike
+§4 3a). A pull that succeeds over an image that then fails to RUN is the bad case, and it is R-443.
+
+### 1.2 The catalog syncer moves the file underneath a deployed app
+
+`Syncer.copyTemplates` (`felhom-controller/controller/internal/sync/sync.go:319`) copies
+`docker-compose.yml` and `.felhom.yml` into **every** stack folder on a 15-minute cycle
+(`internal/config/config.go:351`, default `15m`). **It has no deployed check of any kind.** Its only
+guard is a sha256 content compare in `copyIfChanged` (`sync.go:403`) and its only exclusion is
+`app.yaml`. The post-sync hook is `stackMgr.InjectMissingFields(updated)` and nothing else — **the
+sync does not restart anything.**
+
+That is why a deployed app's compose file and its running containers can disagree **indefinitely**.
+Measured live 2026-09-01: a real catalog pin change travelled the real cycle, the sync rewrote the
+deployed app's file at 17:45:17Z, and the container went on running the old image (spike §3).
+
+### 1.3 Thirteen other paths end in `compose up -d`
+
+Excluding the three API actions, **13 call sites across 9 files** call `StartStack` or `RestartStack`,
+and every one ends in `compose up -d` against the live compose file (spike §8 — the task that
+commissioned the spike said five; the count is thirteen). They include the **boot reconciler**
+(`bootrecon.go:269`), the **app-stop guard's** recovery (`appstop_marker.go:283`), the **drive-return
+gate** (`intermediary.go:222`), the quiesce restart-after-backup, off-site reconstitution, and every
+restore path.
+
+**So an upgrade can happen with nobody pressing anything** — measured, spike §2 variant 1c-ii, where
+a boot reconciliation started an app on a newer image at 17:55:44Z.
+
+### 1.4 One fear is measured SMALLER than it was stated
+
+A plain power cut does **not** upgrade anything. Docker's own `restart: unless-stopped` puts the
+existing containers back on the OLD image, so the boot reconciler finds no orphan and never runs
+`up -d` — it says so in its own words: `no boot-orphaned apps (nothing to start)` (spike §2, variant
+1c, a positive observable and not an absent log line).
+
+**The unattended upgrade needs the narrower precondition: *"and the app did not come back."*** Saying
+so is more useful than leaving the scarier version standing.
+
+---
+
+## 2. What was chosen, and by whom
+
+`Manager.RestartStack` (`internal/stacks/manager.go:1161`) carries this comment, and it predates the
+whole arc:
+
+> *"Use `up -d` instead of bare `restart` so that env vars from app.yaml are injected and any template
+> changes (new images, healthchecks) are picked up. Plain `docker compose restart` only sends
+> SIGTERM+start to existing containers without re-reading the compose file or env."*
+
+**So the restart behaviour was chosen, deliberately, and written down. A design decision is not a
+defect.** What was never decided — and is recorded nowhere — is what happens once the catalog syncer
+moves the file underneath a *deployed* app, and whether the choice was meant to extend to the thirteen
+unattended call sites. **That gap is R-438, and it stays open**: this document records the mechanism;
+it does not change it.
+
+---
+
+## 3. The operator decisions
+
+These are rulings, not proposals. Anything specced against a different assumption is wrong.
+
+### 2026-09-02
+
+1. **The safety copy is a verified recent backup as a PRECONDITION** — not a new copy invented for the
+   update path. The guest-snapshot alternative is to be **spiked before anything is designed around
+   it**. Context: the existing safety machinery (`Manager.writeSafetyDump`,
+   `internal/backup/offbox_reconstitute.go:207`) is **database-only**, which is the headline of spike
+   §6 — the file half was never priced, and demo-hp is too young a box to price it.
+
+2. **The support window runs on HOW FAR BEHIND THE CATALOG a box is, not on how old its version is.**
+   A customer on the newest version is supported however old that version is. This is why
+   `catalog_since` exists and why no version string is shown.
+
+3. **Updates are automatic WITHIN a major, never ACROSS one.** The cross-major case needs a human,
+   because §4 says it cannot be undone.
+
+### 2026-09-06 — **Option 1: freeze the version, keep the fixes flowing.** SHIPPED, v0.235.0
+
+4. **An app's version is frozen to what the customer has, and only a deliberate Update moves it —
+   while corrections to its definition keep arriving on the 15-minute cycle exactly as they do
+   today.**
+
+   **This is the ruling R-447 was blocked on**, and it was blocked for a good reason: §2 establishes
+   that `RestartStack`'s use of `up -d` to pick up template changes was **chosen** and written down in
+   its own comment. Reversing a chosen behaviour is a decision, not a bug fix.
+
+   **What the ruling looked at, and why it is not simply "stop the syncer touching deployed apps".**
+   The old behaviour had two halves and only one of them was unwanted:
+
+   | half | verdict |
+   |---|---|
+   | a restart/repair silently changes the app's VERSION | **unwanted** — nobody chose it, nobody is told, and §4 says it cannot be undone |
+   | template CORRECTIONS reach a deployed app, and a broken definition heals itself within 15 minutes | **worth keeping** — both measured in the spike §3 |
+
+   So the ruling keeps the second and removes the first. In the operator's own words: *while the
+   catalog is offering the same version you are running, its fixes flow to you; the moment it moves to
+   a newer version, you are frozen at what you have until you choose to update.*
+
+   **It does NOT make the Update button safer.** That is slice 4 (R-448), and it is where the backup
+   precondition goes. Slice 3 only stops the other twelve paths from doing the update's job.
+
+---
+
+## 4. The vocabulary ruling — "rollback" is struck
+
+**App data CANNOT be rolled back.** Measured on Nextcloud (spike §7): once a migration has actually
+run, putting the old image tag back produces a container that refuses to start —
+
+> *"the version of the data (32.0.9.2) is higher than the docker image version (31.0.14.1) and
+> downgrading is not supported"*
+
+with a positive control proving the data is intact, only unreachable by the old version (§7 6d).
+
+**So "rollback" must not appear in any spec for this arc.** The two shapes actually available are:
+
+| shape | when it applies | what it does |
+|---|---|---|
+| **ABORT** | before anything migrated | stop, put the old image back, the app runs again |
+| **RESTORE FROM A COPY** | after a migration ran | the data restore is the whole remedy |
+
+There is no third.
+
+### 4.1 MEASURED 2026-09-06 — and the abort turns out to be a property of the APP, not of upgrades
+
+`SPIKE-upgrade-test-2026-09-06.md` upgraded three real apps with real data in them and then attempted
+the abort on each. **All five real catalog upgrades kept the customer's data.** The abort did not
+behave the same way twice:
+
+| app | abort | why |
+|---|---|---|
+| **docmost** `0.25.3`→`0.95.0` | **REFUSES** | the old code finds migration ledger entries it does not know: *"corrupted migrations: previously executed migration 20260213T085259-notifications is missing"*, then *"Failed to run database migration. Exiting program."* |
+| **privatebin** `1.7.5`→`2.0.5` | **works** | file-backed, no database, no schema — a major version moves no data |
+| **bookstack** app+engine | **works, misleadingly** | only because the MariaDB datadir upgrade was skipped and never happened — see R-459 |
+
+**This puts TWO independent measurements behind the ruling above, by two unrelated mechanisms:**
+Nextcloud refused on an explicit version comparison; docmost refuses on its migration ledger. The word
+"rollback" was already struck; it is now struck on evidence rather than on one case.
+
+**And it adds a distinction this document did not have: there is no single answer to "can this update
+be undone". There are apps where it can and apps where it cannot, and the only way to know which is to
+MEASURE THAT APP.** Any design that assumes one answer for all 53 is designing against a fact that was
+checked and is false.
+
+Per §5 below, a restore's image-level undo used to have a ≤15-minute half-life because the syncer
+overwrote it (**R-441**) — closed in v0.235.0.
+
+---
+
+## 5. The shape, as SHIPPED in v0.235.0
+
+The live `docker-compose.yml` is **DERIVED** from a pin recorded in `app.yaml` — the one file the
+syncer never touches. The catalog proposes; `app.yaml` decides; the compose file is an output rather
+than an input, and the thirteen unattended `up -d` paths stop being able to change a version by
+accident.
+
+### 5.1 Nothing was added to the thirteen call sites, and that is deliberate
+
+**They are made safe by removing the reason, not by gating them.** The most important of them are
+REPAIRS — the boot reconciler (`bootrecon.go:269`), the drive-return gate (`intermediary.go:222`),
+the app-stop guard (`appstop_marker.go:283`). **A repair path that refuses to repair leaves a
+customer's app down, which is worse than the problem this slice solves.** Since the file they act on
+no longer changes version, every one of them became safe without being touched.
+
+### 5.2 The pin, and what it is not
+
+`AppConfig.PinnedImages` (`app.yaml`, `pinned_images:`), service → image ref.
+
+**It is NOT `InstalledImages`.** That field is an OBSERVATION — what containers report. This one is a
+DECISION — what should run. Letting an observation feed a decision would make a bad reading become a
+bad deployment, which is the category error `desired_state` exists to avoid (R-166), one field over.
+They will normally agree; when they disagree that is a signal, not a bug to paper over.
+
+**Absent means UNPINNED, and unpinned means the app behaves exactly as it did before v0.235.0.**
+
+Beside it, `applied-compose.yml` in the stack directory stores the exact definition that pin came
+from. `Syncer.copyTemplates` copies exactly `docker-compose.yml` and `.felhom.yml`, so that name is
+safe from the catalog, and keeping it beside the app means it travels with every path that already
+moves a stack dir.
+
+### 5.3 The four writers — the only acts entitled to move a version
+
+| writer | pin source |
+|---|---|
+| the deploy path (`runComposeDeploy`) | the template just deployed from |
+| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull |
+| the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** |
+| `Manager.AdoptPins` | the observation, once, and only when complete AND matching |
+
+**`UpdateStack`'s ordering is load-bearing, not stylistic.** `compose pull` and `up -d` act on the file
+on disk, so the catalog's definition has to BE that file before either runs. A pin set afterwards
+would pull the frozen version and change nothing — while reporting success, and a button that lies is
+worse than a button that refuses. **A failed pin write REFUSES the update**, which is the opposite of
+`recordInstalledImages` and for the same reason `desired_state` refuses: this field is intent.
+
+### 5.4 The render table, complete
+
+| app state | result |
+|---|---|
+| not deployed / protected / seam not wired | the catalog template — today's behaviour |
+| deployed, **unpinned** | the catalog template + one DEBUG |
+| deployed, pinned, catalog images **equal** | the catalog template — **fixes flow, self-healing works** |
+| deployed, pinned, catalog images **differ** | the **stored applied definition** — frozen WHOLE |
+| pinned, differ, nothing stored | the catalog template + one WARN. We cannot freeze what we do not have and must not invent it |
+| mid-deploy | the compose file is left alone this cycle |
+
+**`.felhom.yml` is copied verbatim in every case** — it carries no image, and it carries
+`catalog_since`, which the badge needs. See §8.5.
+
+**The frozen branch writes a WHOLE file and never a substitution.** Taking the new template and
+putting the old refs back creates a third state nobody chose: `wger 2.6` needs a full DB configuration
+the older template cannot supply, so an old image under a new template is broken in a way neither
+version is.
+
+**And this is not "skip deployed apps".** That option was considered and rejected: it also stops
+health-check fixes, memory limits and new deploy fields, and it destroys the self-healing measured in
+the spike §3 — both halves the ruling explicitly kept.
+
+### 5.5 Adoption, and why the startup order matters
+
+`AdoptPins` runs once at boot, immediately after `BackfillInstalledImages`, and pins every deployed app
+to what it is already running. **It reads and writes files only** — no container is started, stopped or
+touched. It skips, loudly, when the observation is incomplete or when the app runs something the
+current template no longer offers; those apps keep pre-v0.235.0 behaviour rather than receive a
+guessed pin.
+
+**`syncer.Start()` was moved to after adoption.** It fires an immediate sync; at its previous position
+that first sync ran while every app was still unpinned, copied the catalog over a deployed app, and
+handed the next restart a version change — the exact behaviour this slice removes, once per boot.
+
+### 5.6 The trap this slice set for the previous one
+
+`Stack.TemplateImages` is read from the app's **live** compose file — which is now the RENDERED one.
+On a frozen app that file names the OLD version, so `web.compareInstalledToTemplate` would find
+installed == template and answer **„Naprakész" on exactly the apps that are behind** — with every test
+still green, because the new field has the same type and shape. The badge now reads
+`Stack.CatalogImages`, taken from the syncer's own git clone. **A feature that silently inverts an
+earlier feature is the failure mode to look for whenever a file changes meaning.**
+
+---
+
+## 6. The seven slices
+
+| # | slice | status |
+|---|---|---|
+| **1** | **The box records what it actually installed** — `app.yaml.installed_images`, per compose service, ref + digest + first-seen. | **SHIPPED, controller v0.233.0 (2026-09-02)** |
+| **1b** | **Seed the record for apps nobody touches** — a startup backfill, so the label is not restricted to apps that happen to get restarted. | **SHIPPED, controller v0.234.0 (2026-09-03)** |
+| **2** | **One badge says whether the app is current** — „Naprakész" / „Frissítés elérhető — N napja", from `catalog_since`. No version number. | **SHIPPED, controller v0.233.0 + catalog `69761cf` (2026-09-02)** |
+| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
+| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | OPEN — R-448 |
+| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
+| **6** | **A version sequence** — updates automatic within a major, a human across one; **an engine change gets its own edge.** | OPEN — R-450 |
+| **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451 |
+
+### The verdict record — the contract Slice 6 carries
+
+Decided here rather than invented twice. The harness writes one of these per edge, beside its
+evidence; Slice 6 puts the same shape in the catalog.
+
+```json
+{"harness_version": 1, "app": "bookstack",
+ "from": {"bookstack": "…:25.02.2", "bookstack-db": "mariadb:11.6"},
+ "to":   {"bookstack": "…:26.05.2", "bookstack-db": "mariadb:12.3"},
+ "verdict": "proven | failed | inconclusive",
+ "seed_read_before": true, "seed_read_after": true, "healthy_after": true,
+ "migration_observed": "verbatim log line, or null",
+ "abort": "starts-and-serves | refuses | starts-data-gone | not-attempted",
+ "abort_detail": "the refusal quoted verbatim, or null",
+ "duration_s": 0, "measured_at": "RFC3339", "evidence": "relative path"}
+```
+
+**`inconclusive` is a first-class verdict and must never be collapsed into `failed`.** "We could not
+measure it" and "it does not work" are different facts, and only one of them is about the app.
+**`migration_observed` is a quoted line, never an inference from timing** — the value of both the
+Nextcloud and the docmost findings was the exact sentence the app printed.
+
+### Database engines under an upgrade — MEASURED 2026-09-06
+
+**The arc's standing rule that an engine change gets its OWN edge now has measured evidence behind
+it**, and the evidence is stronger than the rule's original argument. The rule was justified by
+*"two migrations behind one edge is an unreadable failure when it breaks"* — a readability argument.
+What was measured is that **an engine change can be applied and silently NOT happen**, which the
+app-half edge cannot produce and which no amount of readability would have surfaced:
+
+- `SPIKE-upgrade-test-2026-09-06.md` §4 — MariaDB 12.3 starts on an 11.6 datadir, logs that the
+  conversion it requires was **skipped**, and serves. **Assigned to the engine half by decomposition:**
+  the app half alone produces no such line.
+- `SPIKE-r459-mariadb-upgrade-2026-09-06.md` — it is **stable but never self-resolving** (5 of 5
+  restarts, no degradation, and the engine says `Check required!` every time, forever). Converting
+  properly **succeeds**, costs **7 s**, takes its own system-database backup, and **does not** cost the
+  ability to abort. **The trade that was expected here does not exist.**
+
+**Two rules for anything this arc builds around a database engine:**
+
+1. **Ask the engine, not the log.** MariaDB's entrypoint prints `MariaDB upgrade not required` on an
+   unsupported **downgrade**; `mariadb-upgrade --check-if-upgrade-is-needed` names it exactly
+   (**R-464**). A cheap instrument built on the log line would report "fine" for the broken case.
+2. **The two engines fail in opposite directions, so one check will not do.** MariaDB starts anyway
+   and skips quietly; **PostgreSQL refuses to start** on a datadir from an older major, and the image
+   performs no `pg_upgrade`. Eleven templates carry PostgreSQL and **eight sit on `postgres:16-alpine`**
+   (**R-463**).
+
+**And an engine-state field belongs BESIDE a verdict, never inside it.** `upgrade-test.py` reports
+`engine_state_after` next to `verdict`, because an unconverted datadir is not *known* to be a failure
+and a verdict that said so would encode an unproven judgement.
+
+**The rule slice 6 inherits, recorded now while it is cheap:** an engine change gets its own edge,
+never bundled with an app version bump. `bookstack` moved the application *and* MariaDB 11.6 → 12.3 in
+one commit (`0b73e5e`); that is two migrations behind one edge, and an unreadable failure when it
+breaks.
+
+---
+
+## 7. What slices 1 and 2 actually built
+
+### 7.1 The record (slice 1)
+
+`Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs
+after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`,
+and writes `app.yaml`:
+
+```yaml
+installed_images:
+  web:
+    ref: lscr.io/linuxserver/bookstack:26.05.2
+    digest: sha256:…              # "" if the image was never pulled from a registry
+    at: "2026-09-02T18:41:03Z"    # when this ref+digest was FIRST seen for this service
+```
+
+Three rules, each with its reason:
+
+- **It reads the CONTAINER, never `docker-compose.yml`.** §1.2 is why: that file is the value that has
+  already moved. A record built from it would answer "what will happen next time something runs
+  `up -d`", which is a different question.
+- **A failed write NEVER refuses the action** — deliberately the opposite of `SetDesiredState`.
+  Intent refused, observation logged. Refusing to start a customer's app because we could not write
+  down which version it is trades a real outage for a bookkeeping gap.
+- **It is NOT called from `StartStackServices`** — the R-47 DB-only restore window would overwrite a
+  complete record with a partial one.
+
+**Seeded at startup (v0.234.0).** `Manager.BackfillInstalledImages` runs once at boot, beside the
+desired-state backfill and before the boot reconciler, and records what every deployed app is ALREADY
+on. It only READS containers. **This was not a refinement — without it the feature did not reach a
+quiet box at all:** see §8.3, which was written as a known limitation on 2026-09-02 and was a defect
+by the next morning.
+
+Two admission rules, and the second is the design:
+
+- **It never overwrites an existing record.** The bring-up paths own updates; this fills gaps only.
+- **It refuses to seed a PARTIAL observation.** §7.2's comparison reads a service-count mismatch as
+  BEHIND, so a degraded or crash-looping app seeded from its visible containers would render
+  „Frissítés elérhető" over an app that is perfectly current. The bring-up paths may write a partial
+  because they follow a SUCCESSFUL `up -d`, where a gap is real news and is logged; a backfill meets a
+  box in whatever state it is in. **Same field, two writers, two different admission rules — that is
+  deliberate and must not be "made consistent".**
+
+**Nothing reads it to take a decision.** Slice 2 reads it to render a label.
+
+### 7.2 The label (slice 2)
+
+`web.updateBadge` (`internal/web/updatebadge.go`) compares the recorded reference per service against
+what the current template pins, and renders through the existing `meta_badge` partial — no new markup,
+no new CSS.
+
+**Absent means UNKNOWN and never means current.** Every `app.yaml` written before v0.233.0 has no
+record, so a fall-through to „Naprakész" would have told the whole fleet their months-old apps were
+current. This is the R-166 lesson applied to an observation instead of an intent, and it is pinned by
+a test with a companion red-proof.
+
+**No version number reaches the customer** (operator ruling: a household cannot act on `26.05.2`).
+Version strings stay in the logs, the API and the hub.
+
+---
+
+## 8. Known limitations, stated plainly
+
+1. **„Naprakész" can be FALSE for the 23 floating pins.** The comparison is reference-to-reference and
+   queries no registry — a customer's box must not depend on reaching eight upstream registries to
+   render a page. For `postgres:16-alpine`, `mariadb:11.6` and 21 others the reference can be
+   identical while the image behind it has moved. **Measured, not theorised:** spike §5 found
+   `mariadb:11.4` and `mariadb:12.3` had both already moved upstream, with two fully-pinned controls
+   holding. Digest-level comparison needs a registry query and is deferred — **R-446**.
+2. **Nothing enforces `catalog_since`.** A commit that moves an `image:` line and forgets the date
+   under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
+   commit to diff against, so the gate needs a deeper fetch — **R-452**.
+3. ~~**The record only appears after the next lifecycle action.**~~ **CLOSED in v0.234.0, and the way
+   it closed is worth keeping.** This was written on 2026-09-02 as an accepted limitation — *"the
+   fleet view fills in gradually"*. The operator looked at demo-felhom the next morning and found
+   OpenGist, up 15 hours, running exactly the catalog pin, showing **nothing at all**. **On a quiet
+   box "gradually" means "never", and a feature that fills itself in on an event nobody triggers is,
+   on the quiet installations, not shipped.** `BackfillInstalledImages` now seeds the absences at
+   startup by reading containers (§7.1). **The residue that stays:** the seed happens at controller
+   START, so a box between upgrade and its next restart still shows nothing — bounded by one restart
+   rather than unbounded.
+4. **A frozen app is frozen WHOLE.** While the catalog is ahead, **no** template correction reaches
+   that app — not even one unrelated to the version. That is the direct consequence of §3.4 and of the
+   `wger 2.6` hazard, and it is the right trade: a new template around an old image is a third broken
+   state. Recorded so it is a choice, not a surprise.
+5. **`.felhom.yml` keeps flowing while the compose file is frozen** — the deliberate asymmetry in
+   §5.4. So a frozen app can receive a health check written for a NEWER version and read as degraded.
+   **The failure direction is a false alarm, never data loss**, and freezing `.felhom.yml` would break
+   the update badge by withholding `catalog_since`. **R-458.**
+6. **The Update button is still unguarded.** It takes no backup, has no rollback, and can still
+   attempt a multi-major jump the app will refuse (R-40). **Slice 3 did not change that and must not
+   be read as having done so** — the precondition is slice 4 (R-448).
+7. **An engine major can be applied without its datadir upgrade, and nothing notices.** Measured
+   2026-09-06: the catalog's own bookstack transition starts MariaDB 12.3 on an 11.6 datadir, and the
+   image logs that the required upgrade was **skipped** because the template sets no
+   `MARIADB_AUTO_UPGRADE`. The app serves. Whether that ever breaks is **not** established. **R-459.**
+8. **Only three of 53 apps have ever had an upgrade measured**, and one of them (bookstack) can only
+   be half-proven headlessly (**R-460**). The widening is **R-462**, costed with real numbers.
+9. **The hub does not record image tags at all.** Its report's container payload carries name, state,
+   CPU and memory, and no image field (spike §5). So the fleet view of §6 slice 7 needs a hub-side
+   change; it is not derivable from what is already reported.
+
+---
+
+## 9. Where the rest lives
+
+| what | where |
+|---|---|
+| the measurements this document rests on | `audits/SPIKE-app-update-2026-09-01.md` |
+| the work | `backlog/OPEN-ITEMS.md` — R-438..R-445, R-446..R-452 |
+| the syncer, described accurately but without the consequence | `architecture/02-controller-module-map.md` |
+| what the lifecycle actions are proven to do | `architecture/00-capability-map.md` |
+| the implementation | `felhom-controller/controller/README.md` §"What is installed, and is it current?" |

documentation/architecture/where-felhom-stands.html

@@ -6,8 +6,8 @@
 </style></head><body><div class="wrap">
 <h1>Where Felhom stands</h1>
 <div style="color:#8fa0b8;font-size:12.5px;margin:7px 0 4px;max-width:1100px;line-height:1.5">What we built, what happens when things go wrong, and what is still missing. <b style="color:#cbd7e8">Generated from <code>where-felhom-stands.yaml</code></b>, which cites the capability map, register row or evidence document behind every claim — and which is checked by <code>scripts/check_stands.py</code>.</div>
-<div style="color:#6b7a91;font-size:11.5px;margin-bottom:16px">Verified 2026-08-09 against felhom-agent <code>28ba8593b8</code>, felhom-controller <code>c732fe1283</code>, hub <code>56f8aa611c</code>. <b style="color:#fbbf24">12 status(es) moved in that pass</b> — each is marked on the page.</div>
-<div style="display:flex;gap:22px;flex-wrap:wrap;background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:11px 14px;margin-bottom:18px"><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Walked</b> <span style="color:#7c8aa3">(23)</span><span style="display:block;color:#6b7a91;font-size:10.5px">done end to end on real hardware, evidence on file</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Built</b> <span style="color:#7c8aa3">(14)</span><span style="display:block;color:#6b7a91;font-size:10.5px">shipped and tested, the real path never walked</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Partial</b> <span style="color:#7c8aa3">(14)</span><span style="display:block;color:#6b7a91;font-size:10.5px">some walked, some not — the note says which</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Missing</b> <span style="color:#7c8aa3">(4)</span><span style="display:block;color:#6b7a91;font-size:10.5px">does not exist</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="font-size:11.5px"><b style="color:#7dd3fc">needs-hardware</b><span style="display:block;color:#6b7a91;font-size:10.5px">only a running box could settle it</span></span></div></div>
+<div style="color:#6b7a91;font-size:11.5px;margin-bottom:16px">Verified 2026-08-22 against felhom-agent <code>40d857b527</code>, felhom-controller <code>2da259af38</code>, hub <code>877fcd2a38</code>. <b style="color:#fbbf24">15 claim(s) carry a recorded status move</b> (8 re-checked in this pass) — each is marked on the page.</div>
+<div style="display:flex;gap:22px;flex-wrap:wrap;background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:11px 14px;margin-bottom:18px"><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Walked</b> <span style="color:#7c8aa3">(20)</span><span style="display:block;color:#6b7a91;font-size:10.5px">done end to end on real hardware, evidence on file</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Built</b> <span style="color:#7c8aa3">(14)</span><span style="display:block;color:#6b7a91;font-size:10.5px">shipped and tested, the real path never walked</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Partial</b> <span style="color:#7c8aa3">(17)</span><span style="display:block;color:#6b7a91;font-size:10.5px">some walked, some not — the note says which</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Missing</b> <span style="color:#7c8aa3">(4)</span><span style="display:block;color:#6b7a91;font-size:10.5px">does not exist</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="font-size:11.5px"><b style="color:#7dd3fc">needs-hardware</b><span style="display:block;color:#6b7a91;font-size:10.5px">only a running box could settle it</span></span></div></div>
 <div style="font-size:15px;font-weight:600;margin:0 0 9px">The journey</div>
 <div style="color:#6b7a91;font-size:11.5px;margin-bottom:10px">Seven stages, left to right, as they happen to a person.</div>
 <div class="grid7">
@@ -15,9 +15,9 @@
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">2</span> &nbsp;Making it theirs</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An emailed one-time code; the customer sets their own password and the operator never sees it<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Exercised live 2026-08-09: reset code accepted at /claim, 302 + session.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Customer claim: one-time emailed code → customer sets own p… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer can bind their own machine from a link, with no operator present — done once, for real<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">attempts=0, locked=0, source customer_selfbind, 2026-07-18.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Customer binds their own appliance (self-service) · evidence tests/VALIDATION-n100-rehearsal-2026-07-18.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">It has never been done by a person who is not the operator<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Map records MISSING (as evidence). Still true after 2026-08-09.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map A customer (not the operator) performs a restore via UI alone</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The same word is used for two different secrets across three surfaces; the email points at a page a rebuilt machine does not show<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. Cost a real code on 2026-08-09.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-282 · register R-283</span></span></div></div>
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">3</span> &nbsp;Using it</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Apps installed from a catalog of about 52, with a memory guard and health-aware progress<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">53 templates listed on the rebuilt box 2026-08-09; deploy driven live.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Deploy an app from the catalog (env config, memory guard, h… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Start, stop, restart, update, logs, remove — and the parts that must not be stopped cannot be<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited by the map row or anywhere else</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map App lifecycle: start/stop/restart/update/logs/remove/redeploy · register R-108</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Reachable from anywhere through a tunnel, per-app addresses<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Verified tonight: the rebuilt box answered on its public URL from outside.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Remote access via Cloudflare Tunnel + Traefik (per-app subd… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Reachable on the home network when the internet is down<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#7dd3fc;font-size:10px;white-space:nowrap">needs-hardware</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Only an observation on a running box with WAN pulled could settle it. Boxes are off.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map LAN access when internet is down (lan_resolver)</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Phone photos, documents with text recognition, files from Windows Explorer or a Mac<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Files from Windows Explorer / Mac Finder (SMB server) · evidence audits/SPIKE-lan-discovery-2026-07-18.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A one-tap launcher, and a read-only guest link for visitors<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Indítópult (app launcher) — one-tap grid of the household's…</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-heigLine truncated
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">4</span> &nbsp;Drives</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A new drive is found, offered, formatted, mounted and enrolled — including on awkward older boot layouts<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">moved twice: 2026-08-09 walked -&gt; built, then 2026-08-10 back to walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Applies to a NEW drive. Re-attaching an existing one after a reinstall is R-280 and fails.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: receipt found 2026-08-10: a live throwaway drive on felhom-pve walked scan -&gt; format -&gt; mount -&gt; PVE dir storage -&gt; one-click Regisztralas, with the resulting storage.cfg entry and systemd mount unit recorded. The map already read PROVEN-LIVE; the dataset was behind it. Scope unchanged: this is a NEW drive. Re-attaching an existing one after a reinstall is R-280 and still fails.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Drive wizard: scan/format/mount/enroll, incl. legacy-boot L… · register R-220 · evidence audits/SPIKE-raw-drive-enroll-2026-06-15.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Moving data between drives, crash-safe; removing a drive safely; unplug detected<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Data migration between drives (all / per-app), crash-safe</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A network drive can be browsed and hold bulk media, but may not hold an app's data — enforced<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">RefuseAsAppNamespace is the fail-closed predicate.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Network storage (NAS) is browse + bulk-media only · register R-108 · evidence audits/R108-network-app-namespace-2026-07-30.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">After a reinstall the data drive cannot be re-attached through any dashboard route<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. /api/disks/candidates returns empty; the restore page promises two clicks.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-280 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
-<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">5</span> &nbsp;Backing up</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">App data on the machine, nightly database dumps, a copy on a second drive<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">moved twice: 2026-08-09 walked -&gt; built, then 2026-08-10 back to walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: receipt found 2026-08-10: an adversarial, destructive, unattended overnight campaign across both boxes and ep0, with A2 (one quiesce, two tiers) proven end to end. Map already read PROVEN-LIVE.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Tier-2 secondary-drive copy: class-driven legs · evidence audits/CAMPAIGN-8-backup-restore-2026-07-27.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A whole-machine archive that lands off the guest's own disk — a single-drive machine is recorded as degraded rather than pretending<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Whole-guest backup lands OFF the guest's own physical device · register R-165</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An encrypted off-site copy, sealed with a key the operator cannot read<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Verified tonight from the repo itself: 18 snapshots, daily, unbroken.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Offsite (restic → Hetzner Storage Box) · register R-199 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machines<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">'on both machines, unattended, every tier' is a continuing claim about scheduled runs. Both boxes are off; the last recorded restore-test on demo-hp FAILED (notification_log 2026-08-05 restore_test_failed). Cannot be settled tonight.</span><span style="display:block;color:#c08a3e;font-size:11px;margin-top:2px">proof decay: PROOF-DECAY RULE FIRED (first time it has). A receipt EXISTS - architecture/_recovery-inventory-2026-07-28.md carries live journal lines for scheduled restore-tests on both boxes and both tiers - but it is superseded by later observation: demo-hp logged restore_test_failed on 2026-08-05, and the box has since been wiped and reinstalled (2026-08-09). The claim is about a CONTINUING scheduled behaviour, so a 2026-07-28 observation cannot carry it. Stays grey until a scheduled restore-test is seen passing on the rebuilt box. THE CAPABILITY MAP STILL READS PROVEN-LIVE (2026-08-03) AND IS NOW THE THING OUT OF STEP.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited, and the last recorded restore-test on demo-hp FAILED (2026-08-05)</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">caLine truncated
+<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">5</span> &nbsp;Backing up</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">App data on the machine, nightly database dumps, a copy on a second drive<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">moved twice: 2026-08-09 walked -&gt; built, then 2026-08-10 back to walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">True for every app but one, and that one was found on 2026-08-21: paperless-ngx's 72-table PostgreSQL was dumped nightly and written into a folder named after an app that does not exist, so it never entered the recovery unit, the second-drive copy or the off-site copy. Fixed and proven live 2026-08-22 (R-355) — the dump is now in the app's own unit and in the off-site snapshot for the first time. A catalogue-wide sweep, itself proven able to convict a planted case, says 1 of 53 was affected.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: receipt found 2026-08-10: an adversarial, destructive, unattended overnight campaign across both boxes and ep0, with A2 (one quiesce, two tiers) proven end to end. Map already read PROVEN-LIVE.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Tier-2 secondary-drive copy: class-driven legs · register R-355 · evidence audits/CAMPAIGN-8-backup-restore-2026-07-27.md · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperl…</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A whole-machine archive that lands off the guest's own disk — a single-drive machine is recorded as degraded rather than pretending<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Whole-guest backup lands OFF the guest's own physical device · register R-165</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An encrypted off-site copy, sealed with a key the operator cannot read<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The COPY is real and the key is still unreadable to us — what failed is DAILY and UNBROKEN. The 2026-08-09 note said '18 snapshots, daily, unbroken'; the next snapshot after 2026-08-09 08:30 was 2026-08-21 22:17, put there by hand during the drill. Twelve days, no alarm. Two causes in series: the 2026-08-21 rebuild lost the off-box target (R-193's shape), and after the self-heal restored it EVERY per-app switch was still off, so the first run logged 'backup OK: 0 app(s) backed up, 14s'.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported it</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Offsite (restic → Hetzner Storage Box) · register R-199 · register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The backups prove themselves: a restore is actually performed, uLine truncated
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">6</span> &nbsp;When something goes wrong</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The machine watches itself and repairs some faults without telling anyone it had to<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">R-264 records that the self-heal counters reach the hub and are decoded nowhere.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Box survives an unattended app or guest-network failure · register R-264</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Failures reach the operator by email, one mail per run, every failing app named<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONFIRMED against source: backup_run_failures digest is allowlisted (api/handler.go:1837), operator-only (dispatcher.go:423), templated (templates.go:48); recovery_unit_capture_failed is record-only (dispatcher.go:376). NOTE: R-182's register row still describes the PRE-FIX behaviour — see R-289.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: the digest is wired and source-verified, but no run of it has been observed delivering</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map A failed per-app Tier-1 backup reaches the OPERATOR — EVERY… · register R-182</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Failures reach the customer<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The customer leg is built; today's log shows customer-channel rows skipped as operator_only.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map App crashes → customer notified (one event per transition, …</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An already-paired box is still told to pair itself<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-214 · register R-235</span></span></div></div>
-<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">7</span> &nbsp;Getting everything back</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A rebuilt machine shows a full-page recovery screen without anyone looking for it, and says plainly that nobody can replace a lost recovery code<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Seen unsought on the rebuilt demo-hp 2026-08-09, seal date matching host_escrow.created_at.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer's code opens the sealed package and the data returns byte for byte — including accented Hungarian filenames, verified as raw bytes<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Reproduced 2026-08-09: 4/4 byte-identical, name bytes NFC-preserved, out of snapshot 41c830db.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · evidence tests/walk5-r201-2026-08-07/journal.md · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Walked end to end with no command line inside the machine (2026-08-07)<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#a78bfa;font-size:10px;white-space:nowrap">contested</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONTESTED-RESOLVED: not a contradiction. The walk proves the ROUTE needs no guest shell; the map's MISSING row is about a NON-OPERATOR doing it, which has still never happened. Two questions, one word 'customer'.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · evidence tests/walk5-r201-2026-08-07/journal.md · capability-map A customer (not the operator) performs a restore via UI alone</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Putting restored files back where they belong is still manual<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. Confirmed 2026-08-09: the restore lands in a verification folder and says so.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-213</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The tripwire that says someone is opening this customer's backups does fire<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONFIRMED tonight from the hub store: escrow_blob_served 2026-08-09 10:19:41Z = 12:19 CEST in the operator's mailbox. R-281's original claim of silence is WITHDRAWN.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-281 · register R-285 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
+<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">7</span> &nbsp;Getting everything back</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A rebuilt machine shows a full-page recovery screen without anyone looking for it, and says plainly that nobody can replace a lost recovery code<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Seen unsought on the rebuilt demo-hp 2026-08-09, seal date matching host_escrow.created_at.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer's code opens the sealed package and the data returns byte for byte — including accented Hungarian filenames, verified as raw bytes<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Still true, and re-proven 2026-08-22 (5/5 byte-identical, both accented names as raw bytes) — but the SCOPE is narrower than the sentence sounds and was silently narrower still until v0.218.0. The drill found the off-site restore had NO named-volume leg at all: the archive sat in the unit, the snapshot and the checking folder and was never replayed, under a success message (R-354, fixed and proven 2026-08-22). AND 40 of the 53 catalogue apps STILL cannot run this route at all — it refuses first, saying a running app is not installed (R-356, open). So: proven for an app that declares a data drive; unproven and currently unreachable for the class whose entire dataset is a named volume.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · register R-354 · register R-356 · evidence tests/walk5-r201-2026-08-07/journal.md · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Walked end to end with no command line inside the machine (2026-08-07)<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#a78bfa;font-size:10px;white-space:nowrap">contested</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONTESTED-RESOLVED: not a contradiction. The walk proves the ROUTE needs no guest shell; the map's MISSING row is about a NON-OPERATOR doing it, which has still never happened. Two questions, one word 'customer'.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · evidence tests/walk5-r201-2026-08-07/journal.md · capability-map A customer (not the operator) performs a restore via UI alone</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Putting restored files back where they belong is still manual<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. Confirmed 2026-08-09: the restore lands in a verification folder and says so.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-213</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The tripwire that says someone is opening this customer's backups does fire<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONFIRMED tonight from the hub store: escrow_blob_served 2026-08-09 10:19:41Z = 12:19 CEST in the operator's mailbox. R-281's original claim of silence is WITHDRAWN.</span><span style="display:block;color:#4b5bLine truncated
 </div>
 <div style="font-size:15px;font-weight:600;margin:22px 0 9px">When things go wrong</div>
 <div style="color:#6b7a91;font-size:11.5px;margin-bottom:10px">19 situations, and what actually happens in each.</div>
@@ -28,7 +28,7 @@
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">An app crashes</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An app crashes — the email leg has never been confirmed end to end<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Consistent with fault.operator-email: the digest is wired but its delivery is unobserved.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map App crashes → customer notified (one event per transition, …</span></span></div></div>
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">Power cut mid-backup</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Power cut mid-backup<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/AUDIT-power-outage-recovery-2026-07-22.md</span></span></div></div>
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The guest is destroyed</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The guest is destroyed<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · evidence audits/DRILL-r201-night-run-2026-08-04.md · evidence audits/DRILL-r201-night-run-2026-08-04.md</span></span></div></div>
-<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The whole machine is wiped and reinstalled</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The whole machine is wiped and reinstalled — the data comes back<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">4/4 byte-identical 2026-08-09.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
+<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The whole machine is wiped and reinstalled</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The whole machine is wiped and reinstalled — the data comes back<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The 2026-08-09 rehearsal really did return 4/4 byte-identical, and that stands. What the next real reinstall showed (2026-08-21, demo-hp) is that the rebuild ORPHANS BOTH OFF-PREMISES TIERS at once, quietly: the restic target was lost and needed a self-heal plus a per-app re-enable before any copy resumed (R-193), and the PBS archives from before the reinstall cannot be opened by the rebuilt box at all, because it now presents a different key (R-366). The data came back in the rehearsal because the rehearsal restored it immediately; a machine left alone after a reinstall is not protected in the meantime and nothing says so.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: a real reinstall on 2026-08-21 left both off-premises tiers broken - one silently for 12 days, the other unreadable - so 'the data comes back' holds only if someone restores it at once</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-193 · register R-366 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div></div>
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The whole machine is wiped and reinstalled</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The whole machine is wiped and reinstalled — the journey needs a terminal twice<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONTESTED-RESOLVED against the map: the map's PROVEN-LIVE row is scoped to a controller-data-volume REBUILD (2026-08-04), not a whole-host reinstall. The map has no row for the host case, so there was no contradiction — only a gap.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-273 · register R-280 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The machine is stolen, and someone opens the backups</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The machine is stolen, and someone opens the backups — the operator is told<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Confirmed 2026-08-09 from the store and the mailbox.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-281 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The customer forgets their dashboard password</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer forgets their dashboard password<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Exercised 2026-08-09 via the reset code.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
@@ -40,7 +40,7 @@
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A fresh install picks up an old image</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A fresh install picks up an old image<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Narrowed 2026-08-09: the resume path fetched the vouched golden; a fresh install still takes the newest LOCAL archive with no manifest comparison.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-274</span></span></div></div>
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The off-site provider account is deleted</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The off-site provider account is deleted<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The credential that holds the customer's documents can still delete.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-95</span></span></div></div>
 <div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The machine is switched off for an afternoon</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The machine is switched off for an afternoon — no notion of expected downtime<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Confirmed 2026-08-09: eight operator mails for deliberate, attended work.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-285</span></span></div></div>
-<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A customer restores their own data with no help</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A customer restores their own data with no help<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#a78bfa;font-size:10px;white-space:nowrap">contested</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONTESTED-RESOLVED: the map's MISSING is about a NON-OPERATOR performing it; the walks prove the route, not the person. Neither record was wrong.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map A customer (not the operator) performs a restore via UI alone · register R-201</span></span></div></div>
+<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A customer restores their own data with no help</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A customer restores their own data with no help<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#a78bfa;font-size:10px;white-space:nowrap">contested</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONTESTED-RESOLVED: the map's MISSING is about a NON-OPERATOR performing it; the walks prove the route, not the person. Neither record was wrong. SINCE 2026-08-21 there is a second, harder blocker and it is not about the person at all: for the 40 of 53 apps that declare no data drive the off-site restore REFUSES before it starts, telling the customer a running app 'nincs telepítve' and to reinstall it to the same place — which those apps give them no way to choose (R-356). Those are exactly the apps whose whole dataset is a named volume. Until that is fixed, most customers cannot self-restore off-site even in principle.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map A customer (not the operator) performs a restore via UI alone · register R-201 · register R-356 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div></div>
 </div>
 <div style="margin-top:26px;padding-top:12px;border-top:1px solid #1f2c44;color:#4b5b76;font-size:10.5px;line-height:1.6">Generated by <code>scripts/render_stands.py</code> from <code>where-felhom-stands.yaml</code>. Do not hand-edit this file — edit the data and regenerate. The 2026-08-09 React bundle is kept as <code>where-felhom-stands-2026-08-09-snapshot.html</code> and is not maintained.</div>
 </div></body></html>

documentation/architecture/where-felhom-stands.yaml

@@ -12,11 +12,11 @@
 # verdict vocabulary: confirmed | downgraded | upgraded | contested | needs-hardware
 # depth: source-read (opened live source or evidence) | register+map (checked against the
 #        register and capability map only) | needs-hardware (cannot be settled off-box)
-verified_on: 2026-08-09
+verified_on: 2026-08-22
 verified_against:
-  felhom-agent: 28ba8593b8
-  felhom-controller: c732fe1283
-  hub: 56f8aa611c
+  felhom-agent: 40d857b527
+  felhom-controller: 2da259af38
+  hub: 877fcd2a38
 claims:
   - id: install.iso-selfregister
     band: journey
@@ -301,17 +301,20 @@ claims:
     stage: 5
     title: "App data on the machine, nightly database dumps, a copy on a second drive"
     status: walked
+    note: "True for every app but one, and that one was found on 2026-08-21: paperless-ngx's 72-table PostgreSQL was dumped nightly and written into a folder named after an app that does not exist, so it never entered the recovery unit, the second-drive copy or the off-site copy. Fixed and proven live 2026-08-22 (R-355) — the dump is now in the app's own unit and in the off-site snapshot for the first time. A catalogue-wide sweep, itself proven able to convict a planted case, says 1 of 53 was affected."
     sources:
       - capability-map: "Tier-2 secondary-drive copy: class-driven legs"
+      - register: "R-355"
       - evidence: "audits/CAMPAIGN-8-backup-restore-2026-07-27.md"
+      - evidence: "audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperless/FINDING.txt"
     changed:
       from: built
       also_moved: 2026-08-09 walked -> built
       reason_superseded: "no walk document cited"
       reason: "receipt found 2026-08-10: an adversarial, destructive, unattended overnight campaign across both boxes and ep0, with A2 (one quiesce, two tiers) proven end to end. Map already read PROVEN-LIVE."
     verified:
-      date: 2026-08-09
-      verdict: upgraded
+      date: 2026-08-22
+      verdict: confirmed
       depth: source-read
   - id: backup.whole-machine
     band: journey
@@ -332,59 +335,75 @@ claims:
     band: journey
     stage: 5
     title: "An encrypted off-site copy, sealed with a key the operator cannot read"
-    status: walked
-    note: "Verified tonight from the repo itself: 18 snapshots, daily, unbroken."
+    status: partial
+    note: "The COPY is real and the key is still unreadable to us — what failed is DAILY and UNBROKEN. The 2026-08-09 note said '18 snapshots, daily, unbroken'; the next snapshot after 2026-08-09 08:30 was 2026-08-21 22:17, put there by hand during the drill. Twelve days, no alarm. Two causes in series: the 2026-08-21 rebuild lost the off-box target (R-193's shape), and after the self-heal restored it EVERY per-app switch was still off, so the first run logged 'backup OK: 0 app(s) backed up, 14s'."
     sources:
       - capability-map: "Offsite (restic → Hetzner Storage Box)"
       - register: "R-199"
+      - register: "R-193"
       - evidence: "audits/REHEARSAL-byo-reinstall-2026-08-09.md"
+      - evidence: "audits/REPORT-DRILL-backup-truth-2026-08-21.md"
+    changed:
+      from: walked
+      reason: "the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported it"
     verified:
-      date: 2026-08-09
-      verdict: confirmed
+      date: 2026-08-22
+      verdict: downgraded
       depth: source-read
   - id: backup.restore-proof
     band: journey
     stage: 5
     title: "The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machines"
     status: built
-    note: "'on both machines, unattended, every tier' is a continuing claim about scheduled runs. Both boxes are off; the last recorded restore-test on demo-hp FAILED (notification_log 2026-08-05 restore_test_failed). Cannot be settled tonight."
+    note: "STILL GREY, and for a sharper reason than in August. The scheduler runs and the check works — it failed again on 2026-08-21, unprompted, and named the cause exactly: after the reinstall the box presents a different PBS key than its own older archives were sealed with, so those archives cannot be opened at all (R-366). A tier whose archives are orphaned is a different alarm from a tier whose test failed, and only the second is being said."
     sources:
       - capability-map: "Restore-proof is UNATTENDED — the scheduler covers EVERY tier"
       - register: "R-86"
+      - register: "R-366"
+      - evidence: "audits/REPORT-DRILL-backup-truth-2026-08-21.md"
     changed:
       from: walked
       reason: "no walk document cited, and the last recorded restore-test on demo-hp FAILED (2026-08-05)"
       decay: "PROOF-DECAY RULE FIRED (first time it has). A receipt EXISTS - architecture/_recovery-inventory-2026-07-28.md carries live journal lines for scheduled restore-tests on both boxes and both tiers - but it is superseded by later observation: demo-hp logged restore_test_failed on 2026-08-05, and the box has since been wiped and reinstalled (2026-08-09). The claim is about a CONTINUING scheduled behaviour, so a 2026-07-28 observation cannot carry it. Stays grey until a scheduled restore-test is seen passing on the rebuilt box. THE CAPABILITY MAP STILL READS PROVEN-LIVE (2026-08-03) AND IS NOW THE THING OUT OF STEP."
+      worse_2026_08_22: "It failed AGAIN, unprompted, on 2026-08-21 21:59 - and the cause is worse than 'untested'. Hub event 3016: the PBS archive of 2026-08-18 could not be restored because the manifest's key does not match the key the rebuilt box now presents. The archives that predate the 2026-08-21 reinstall are UNREADABLE to the machine that made them (R-366). Credit where due: the mechanism caught it and named the key mismatch precisely. The gap is that it is reported as 'a restore test failed' rather than 'your older whole-machine backups cannot be opened on this box'."
     verified:
-      date: 2026-08-09
+      date: 2026-08-22
       verdict: downgraded
       depth: needs-hardware
   - id: backup.fill-warning
     band: journey
     stage: 5
     title: "The customer is warned before a drive fills, per drive, in their own language"
-    status: walked
-    note: "R-177 (no operator-triggerable run) limits testing, not the capability."
+    status: partial
+    note: "The warning is real and was SEEN firing on 2026-08-21 with the right Hungarian copy, naming the drive and the free space. What 'BEFORE' cannot survive is the cadence: the watcher runs once a day at 03:30 plus once at startup (R-363), so a filesystem that fills at 03:31 goes unannounced for ~24 h. Watched live: the 69 GB volume carrying all 40-class app data was filled to 99% and the watcher said nothing, while the backup reserve was already refusing an app per run and telling the hub about it."
     sources:
       - capability-map: "The customer is warned BEFORE a filesystem fills"
       - register: "R-167"
+      - register: "R-363"
       - evidence: "audits/SPIKE-r165-mp1-merge-2026-08-02.md"
+      - evidence: "audits/REPORT-DRILL-backup-truth-2026-08-21.md"
+    changed:
+      from: walked
+      reason: "a daily check cannot carry the word BEFORE; observed silent for the whole window a filesystem sat at 99%"
     verified:
-      date: 2026-08-09
-      verdict: confirmed
-      depth: register+map
+      date: 2026-08-22
+      verdict: downgraded
+      depth: source-read
   - id: backup.sikeres
     band: journey
     stage: 5
     title: "A backup that covered nothing still calls itself successful"
     status: partial
-    note: "Warning card."
+    note: "Warning card — and the drill found two more of it, both live. (1) An off-site run with no app selected logs 'backup OK: 0 app(s) backed up'; the card does say 'nincs kijelölt alkalmazás' beside the green tick, so this one is honest if you read past the tick. (2) A restore that placed nothing reported success: '0 fájl visszaállítva', ok=true. The second is the one that matters and it is R-353, still open. The volume half of it is fixed (R-354): the message now names what came back."
     sources:
       - register: "R-240"
+      - register: "R-353"
+      - register: "R-354"
+      - evidence: "audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experiment/messages-verbatim.txt"
     verified:
-      date: 2026-08-09
+      date: 2026-08-22
       verdict: confirmed
-      depth: register+map
+      depth: source-read
   - id: fault.selfheal
     band: journey
     stage: 6
@@ -460,13 +479,16 @@ claims:
     stage: 7
     title: "The customer's code opens the sealed package and the data returns byte for byte — including accented Hungarian filenames, verified as raw bytes"
     status: walked
-    note: "Reproduced 2026-08-09: 4/4 byte-identical, name bytes NFC-preserved, out of snapshot 41c830db."
+    note: "Still true, and re-proven 2026-08-22 (5/5 byte-identical, both accented names as raw bytes) — but the SCOPE is narrower than the sentence sounds and was silently narrower still until v0.218.0. The drill found the off-site restore had NO named-volume leg at all: the archive sat in the unit, the snapshot and the checking folder and was never replayed, under a success message (R-354, fixed and proven 2026-08-22). AND 40 of the 53 catalogue apps STILL cannot run this route at all — it refuses first, saying a running app is not installed (R-356, open). So: proven for an app that declares a data drive; unproven and currently unreachable for the class whose entire dataset is a named volume."
     sources:
       - register: "R-201"
+      - register: "R-354"
+      - register: "R-356"
       - evidence: "tests/walk5-r201-2026-08-07/journal.md"
       - evidence: "audits/REHEARSAL-byo-reinstall-2026-08-09.md"
+      - evidence: "audits/REPORT-DRILL-backup-truth-2026-08-21.md"
     verified:
-      date: 2026-08-09
+      date: 2026-08-22
       verdict: confirmed
       depth: source-read
   - id: recover.no-shell
@@ -581,13 +603,19 @@ claims:
   - id: fail.wiped-reinstalled.data
     band: failures
     title: "The whole machine is wiped and reinstalled — the data comes back"
-    status: walked
-    note: "4/4 byte-identical 2026-08-09."
+    status: partial
+    note: "The 2026-08-09 rehearsal really did return 4/4 byte-identical, and that stands. What the next real reinstall showed (2026-08-21, demo-hp) is that the rebuild ORPHANS BOTH OFF-PREMISES TIERS at once, quietly: the restic target was lost and needed a self-heal plus a per-app re-enable before any copy resumed (R-193), and the PBS archives from before the reinstall cannot be opened by the rebuilt box at all, because it now presents a different key (R-366). The data came back in the rehearsal because the rehearsal restored it immediately; a machine left alone after a reinstall is not protected in the meantime and nothing says so."
     sources:
+      - register: "R-193"
+      - register: "R-366"
       - evidence: "audits/REHEARSAL-byo-reinstall-2026-08-09.md"
+      - evidence: "audits/REPORT-DRILL-backup-truth-2026-08-21.md"
+    changed:
+      from: walked
+      reason: "a real reinstall on 2026-08-21 left both off-premises tiers broken - one silently for 12 days, the other unreadable - so 'the data comes back' holds only if someone restores it at once"
     verified:
-      date: 2026-08-09
-      verdict: confirmed
+      date: 2026-08-22
+      verdict: downgraded
       depth: source-read
   - id: fail.wiped-reinstalled.journey
     band: failures
@@ -727,11 +755,13 @@ claims:
     band: failures
     title: "A customer restores their own data with no help"
     status: partial
-    note: "CONTESTED-RESOLVED: the map's MISSING is about a NON-OPERATOR performing it; the walks prove the route, not the person. Neither record was wrong."
+    note: "CONTESTED-RESOLVED: the map's MISSING is about a NON-OPERATOR performing it; the walks prove the route, not the person. Neither record was wrong. SINCE 2026-08-21 there is a second, harder blocker and it is not about the person at all: for the 40 of 53 apps that declare no data drive the off-site restore REFUSES before it starts, telling the customer a running app 'nincs telepítve' and to reinstall it to the same place — which those apps give them no way to choose (R-356). Those are exactly the apps whose whole dataset is a named volume. Until that is fixed, most customers cannot self-restore off-site even in principle."
     sources:
       - capability-map: "A customer (not the operator) performs a restore via UI alone"
       - register: "R-201"
+      - register: "R-356"
+      - evidence: "audits/REPORT-DRILL-backup-truth-2026-08-21.md"
     verified:
-      date: 2026-08-09
+      date: 2026-08-22
       verdict: contested
       depth: source-read

documentation/audits/AUDIT-gate-decoys-2026-09-01.md

@@ -0,0 +1,81 @@
+# AUDIT — can this check be fooled by a label? The decoy sweep (2026-09-01, R-421)
+
+**Question asked of every gate:** *could I satisfy this with a convincing label instead of the real
+thing?* Answered by construction — a decoy is written, the gate is run, and the verdict recorded.
+**No row's last column was answered from reading.**
+
+## The survey
+
+**29 distinct scripts, 35 registrations** (`reuse-refs`, `instructions`, `observations` are one
+script each, registered in three runners). This matches the task's count of 29.
+
+| gate | runner(s) | meant to prove | actually matches | shape | decoy passed **before**? |
+|---|---|---|---|---|---|
+| emoji | ctrl | no emoji in UI copy | codepoints, in `listdir` scope | 1 | **YES** |
+| native-confirm | ctrl | no OS-modal dialogs | JS call regex, `listdir` scope | 1 | **YES** |
+| app-row-dedup | ctrl | one row markup | regex + `'…' not in src`, `listdir` | 1, 2 | **YES** (×2) |
+| template-id | ctrl | JS ids resolve | id sets, `listdir` scope | 1 | **YES** |
+| secret-markup | ctrl | no secret in markup | template actions, `listdir` | 1 | **YES** |
+| retrieval-promise | ctrl | promises are registered | stems, `listdir` scope | 1 | **YES** |
+| hub-confirm | eu | no OS-modal dialogs | JS call regex, `listdir` scope | 1 | **YES** |
+| manifest-bearer | eu | no bearer literals | 64-hex regex, `listdir` scope | 1 | **YES** |
+| observations | eu, ctrl, agent | a finding is filed | `FILED:` anywhere in the body | 2 | **YES** (R-419) |
+| debug-routes | ctrl | controls resolve | raw text, comments included | 2, 3 | **YES** |
+| closed-register | eu | closed rows are closed | verdict cell — **but skipped unparseable rows** | 2 | **YES** |
+| site | eu | pages are well-formed | a 7-entry `PAGES` list | 1 | **YES → R-423** |
+| one-register | eu | open work is registered | the state cell; `idea` escapes | 2 | **YES → R-424** |
+| offbox-rename | ctrl | branding is retired | a fixed 3-entry `FILES` list | 1 | **YES → R-425** |
+| reuse-refs | eu, ctrl, agent | citations resolve | paths — only 7 extensions | 1 | **YES → R-422** |
+| mojibake | ctrl | no double-encoded UTF-8 | bytes, via `os.walk` | 5 | NO — **control** |
+| docker-v | ctrl | `-v` mounts are safe | argv, via `os.walk` | 5 | NO |
+| image-pins | catalog | no floating tags | real `image:` refs | 5 | NO |
+| golden-currency | eu | a golden was baked | `GOLDEN_SHA256` in the bake log | 5 | NO (R-410 fixed) |
+| golden-notice | ctrl | ditto, other direction | imports the gate above | 5 | NO |
+| instructions | eu, ctrl, agent | instruction files stay sane | effective text | 5 | NO |
+| hub-copy | eu | retired names are gone | `os.walk` over hub/internal | 5 | NO |
+| release-complete | agent | a release is complete | git tag + ancestry + HTTP HEAD | 5 | NO |
+| hostinstall | eu | installer invariants | — | ? | **UNKNOWN** |
+| wire-contract | eu | emitted fields decode | — | ? | **UNKNOWN** |
+| due-checks | eu | dated checks fire | — | ? | **UNKNOWN** |
+| published | agent | versions are published | — | ? | **UNKNOWN** |
+| image-resolvable | catalog | images exist | `docker manifest inspect` | 5 | **UNKNOWN** |
+| volume-persistence | catalog | data survives | runs containers, diffs | 5 | **UNKNOWN** |
+
+**19 sound · 4 holes left open with rows · 6 unknown.** Ten holes were fixed in this session.
+
+## The one cause behind eight of them
+
+Eight gates decided their **scope** with `os.listdir`, one directory level. No template or manifest
+subdirectory exists today, so every one was green **and correct** — and would have stayed green the
+moment anyone added `templates/partials/`, which is an ordinary act. `mojibake` and `docker-v`
+already used `os.walk`, caught the identical planted file, and are the **control that proves the
+cause was the listing and not the decoy.**
+
+## Holes left open, with rows
+
+| row | gate | why not fixed here |
+|---|---|---|
+| **R-422** | reuse-refs | `PATH_RE` matches 7 extensions; a rotted `.md` citation is invisible. Widening it needs a false-positive pass over 4 repos. |
+| **R-423** | site | `PAGES` is a hardcoded list of 7. Fix is to glob `website/*.html`, which needs the per-page exemptions rethought. |
+| **R-424** | one-register | a defect parked under state `idea` escapes. Declared in the gate's own docstring as residual hole 1. |
+| **R-425** | offbox-rename | fixed 3-entry `FILES` list; a new offbox file is unscanned. |
+| **R-426** | — | owns the meta-gate's 20-name exemption list. |
+
+Each is asserted **as it behaves today** where a test could hold it, so the day it is fixed the
+assertion fails and is updated deliberately. A hole nothing asserts is a hole nobody remembers.
+
+## Decoys withdrawn as illegitimate — mine, and named
+
+§2.1 sets the standard: a decoy nobody would write proves nothing. These were withdrawn rather than
+counted, because counting them would have manufactured findings.
+
+1. **image-pins / `x-image: nginx:latest`** — `x-` fields are inert in Compose. The label had no fact
+   behind it *either way*. Replaced with a positive control (a real untagged `image:` line → convicted).
+2. **release-complete / a non-version heading on top of CHANGELOG.md** — `HEAD_RE.search` scans the
+   whole file, so the real release is still found and named `v0.130.0`. The gate is sound.
+3. **app-row-dedup / one commented-out partial call** — `dashboard.html` has **two**; replacing one
+   left the other live. The gate was right and my decoy was half-built.
+4. **one-register / a 3-column decoy row** — it convicted for a *structural* reason, not the one under
+   test. Rebuilt at the real 5-column shape, and the hole then reproduced.
+5. **template-id, secret-markup, retrieval-promise, hub-copy / wrong content** — my first planted
+   files carried nothing those gates hunt for, so "passed" meant nothing. Rebuilt with real triggers.

documentation/audits/DIAG-smart-passed-trap-2026-08-14.md

@@ -0,0 +1,196 @@
+# DIAG — the SMART `PASSED` trap, and the disk alert that reached nobody
+
+**Date:** 2026-08-14
+**Drive:** Seagate `ST3000VX010-2E3166`, S/N `Z6A07P2G`, 3.0 TB, `/dev/sdg` on **DooPlex**
+**Fixtures:** `fixtures/smart-ST3000VX010-failing-2026-08-14.json` (raw `smartctl -a -j /dev/sdg`),
+`fixtures/smartd-history-sdg-2026-08-14.txt` (406 `smartd` journal lines for this device, 11–14 Aug)
+**Status:** the three controller defects named here are FIXED in controller **v0.215.0**; the
+counterfactual in §5 is derived from source, **not** reproduced live.
+
+This is the project's first genuinely failing disk. Before it, the capability map recorded the
+disk-failure scenario as *"Healthy path only — a genuinely failing disk has never been seen."*
+
+---
+
+## 1. What happened
+
+On **11 August 12:28** a 3 TB drive in DooPlex reported its first unreadable sectors. By **13 August
+21:58** it was at 360, and it was taking down a running service. Throughout the entire episode the
+drive's own overall self-assessment read **`PASSED`**, and it still does.
+
+`smartd` was running the whole time and mailed **local root** — a mailbox nobody reads. The failure
+was actually found by a crashlooping pod, not by any alert.
+
+---
+
+## 2. The mechanism — `smart_status.passed` cannot fail on unreadable sectors
+
+Measured, from the committed fixture:
+
+| ID  | Attribute                | value | worst | thresh | raw    |
+|-----|--------------------------|-------|-------|--------|--------|
+| 5   | `Reallocated_Sector_Ct`  | 100   | 100   | **10** | 0      |
+| 187 | `Reported_Uncorrect`     | **1** | **1** | **0**  | **1001** |
+| 188 | `Command_Timeout`        | 100   | 100   | **0**  | 0      |
+| 197 | `Current_Pending_Sector` | 98    | 98    | **0**  | **352** |
+| 198 | `Offline_Uncorrectable`  | 98    | 98    | **0**  | **352** |
+| 199 | `UDMA_CRC_Error_Count`   | 200   | 200   | **0**  | 0      |
+
+`smart_status.passed` is false only when some attribute's **normalized value** falls **at or below**
+its **threshold**. Attributes 187, 197 and 198 — the three that record unreadable sectors — all carry
+`thresh: 0`. A normalized SMART value floors at 1 and cannot reach 0.
+
+> **Therefore `smart_status.passed` is structurally incapable of failing on unreadable sectors.**
+> This is not a quirk of this drive or this vendor. It is how a zero threshold behaves.
+
+Attribute **187 `Reported_Uncorrect` sits at normalized `1` against threshold `0`** with a raw count
+of **1001**. It is one point from failing and has no remaining point to give.
+
+Corroborating, from the same fixture: `ata_smart_error_log.summary.count = 1001`, power-on hours
+**60505** (~6.9 years), `smartctl` exit status **64** (bit 6 — *the device error log contains
+records of errors*) while `smart_status.passed` is still `true`.
+
+**Any monitor built on the drive's overall verdict is blind to this entire class of failure.**
+Felhom already knows better — `agentapi.DiskVerdictFor` reads the raw counters — which is why it
+would have noticed on 11 August, two days early.
+
+---
+
+## 3. Unreadable sectors are not monotonic
+
+From `smartd-history-sdg-2026-08-14.txt`, `Current_Pending_Sector` over the episode (30-minute
+sampling, host clock = CEST):
+
+```
+Aug 11 12:28   8      first sighting
+Aug 11 12:58  16      (+8)
+Aug 11 13:28   0      FULL CLEAR — "No more Currently unreadable (pending) sectors,
+                       warning condition reset after 1 email"
+Aug 11 20:28   8      returns
+Aug 12 01:28  16  →  01:58   8  (-8)
+Aug 12 02:28  32  →  02:58  24  (-8)
+Aug 12 03:28  24      187 Reported_Uncorrect 100→97; ATA error count 0→3
+Aug 12 03:58  16  (-8)  →  04:28  24  (+8)
+Aug 12 10:58  32      … steady 32 for ~11h …
+Aug 12 21:58  24  (-8)
+Aug 13 03:28  40  (+16)  →  04:28  24  (-16)
+Aug 13 11:28  64      terminal run begins — never returns below 64
+Aug 13 11:58  72   12:58  80   13:28 112   15:58 120
+Aug 13 21:58 360      (+240)
+Aug 13 22:28 352  (-8)   … steady 352 through 14 Aug …
+```
+
+Two measured facts carry design weight:
+
+1. **The 11 August excursion cleared completely within one hour** (12:28 → 13:28). A bare `> 0`
+   alarm would have fired on a drive that then looked fine for seven hours. This is why the ladder
+   uses a *sustain* rule rather than a bare non-zero test.
+2. **The benign excursion peaked at 16; the terminal run crossed 64 at 13 Aug 11:28 and never came
+   back.** That is the entire empirical basis for the static count threshold of 64 — see §6.
+
+---
+
+## 4. The three controller defects
+
+All three are in `felhom-controller` at `3e3ee94` (v0.214.0), the tree audited here.
+
+### D1 — the alert carries a severity the hub does not recognise *(highest value)*
+
+`internal/notify/notifier.go:565` — `NotifyDiskHealthDegraded` sets:
+
+```go
+severity := "warn"
+```
+
+The hub accepts an exact-match lowercase vocabulary and **coerces anything else to `info`**:
+
+- `felhom.eu/hub/internal/api/handler.go:2121-2126` — `case "info", "warning", "error", "critical":`
+  … `default: payload.Severity = "info"`
+- `felhom.eu/hub/internal/notify/dispatcher.go:89-96` — `severityNotifies` returns true only for
+  `warning` / `error` / `critical`.
+
+`"warn"` is not in the accepted set. So the Figyelmeztetés-level disk alert is **stored as an
+informational notice and emailed to nobody**, on the customer leg and the operator leg alike.
+
+The function's own doc comment reads *"The hub applies its own per-event-type cooldown"* — which
+presumes it routes. An invariant asserted in a comment with no test pinning it; this project's
+recurring shape.
+
+### D2 — no level above "worth keeping an eye on it"
+
+`internal/agentapi/diskverdict.go:28-41` returns `Warn` for *any* non-zero counter, and can only
+reach `Fail` when `Health == "FAILING"` — which, by §2, this fault class cannot produce. A drive with
+one aging sector and a drive at 352 unreadable sectors rendered the identical chip and the identical
+mild sentence.
+
+### D3 — it speaks once, and forgets on restart
+
+`internal/web/disk_health.go:182` emits only on `v > prev`, against a **in-memory** baseline
+(`disk_health.go:26-29`, *"Lost on restart → the next check re-baselines silently"*). Consequences:
+
+- Between 8 and 352 pending sectors the verdict never changes level, so **nothing further is emitted**.
+- A controller restart while a disk is already bad re-baselines it silently — that disk never alerts
+  again.
+
+---
+
+## 5. Counterfactual — what a customer would have received
+
+**Derived from the source above plus the §3 timeline. NOT reproduced live.**
+
+| Date/time | Drive state | Felhom verdict at v0.214.0 | Emitted | Delivered |
+|-----------|-------------|-----------------------------|---------|-----------|
+| 11 Aug 12:28 | pending 8 | OK → Figyelmeztetés | `disk_health_degraded`, severity `warn` | **nothing** — coerced to `info`, dropped by `severityNotifies` |
+| 11 Aug 13:28 | pending 0 | Figyelmeztetés → Rendben | none (recovery is silent) | nothing |
+| 11 Aug 20:28 → 13 Aug | 8 → 352 | Figyelmeztetés throughout | none (no level change) | nothing |
+| 13 Aug 21:58 | pending 360 | Figyelmeztetés | none | nothing |
+
+> **Felhom would have emitted zero emails about this drive.** The one event it did produce was filed
+> at `info` and delivered to no one.
+
+Note the two defects compound: even had D1 been fixed alone, the customer would have received a
+single mild "Javasolt figyelemmel kísérni" at 8 sectors on 11 August and then silence through 352.
+
+---
+
+## 6. Provenance of the thresholds chosen in v0.215.0
+
+- **64 unreadable sectors → Hiba.** The observed benign excursion peaked at **16** and cleared inside
+  an hour; the terminal run passed **64** at 13 Aug 11:28 and never returned below it. 64 sits above
+  the one observed transient and below the observed terminal run. **This is a judgement from ONE
+  drive.** It is a static backstop and is expected to be replaced in Phase 3 by growth-rate detection
+  over real history.
+- **Sustain before count.** The primary rule is "unreadable sectors still present at the next check";
+  the count is the backstop. On this drive sustain fires **12 Aug**, the count not until **13 Aug** —
+  a full day earlier. The backstop exists for a box that was powered off or restarted across the
+  sustain window.
+- **55 / 60 °C.** Adopted unchanged from the operator's existing Prometheus bands on DooPlex, so the
+  two systems cannot disagree about the same drive.
+
+---
+
+## 7. Measured vs inferred
+
+**Measured** (reproducible from the committed fixtures):
+- The attribute table, thresholds and raw values in §2; `passed: true` at 352 pending sectors.
+- The full non-monotonic timeline in §3, including the one-hour full clear.
+- The three defect locators in §4 — read from live source on both the controller and the hub side.
+
+**Inferred** (source-derived, not executed):
+- The §5 counterfactual. It follows from the §4 locators and the §3 timeline; **no email path was
+  exercised against this drive**, and no `disk_health_degraded` event for it exists in the hub.
+
+**Not covered here:**
+- Attributes **187**, **199** and **188** are not on the agent→controller wire today. Adding them is
+  Phase 2 (a declared wire change, so the hub models them in the same session under G-1). Everything
+  the v0.215.0 fix needs was already on the wire.
+- The new Fail-from-counters path has **not** fired on real hardware — only against this fixture's
+  values in unit tests.
+
+---
+
+## 8. One layer out
+
+`smartd` on DooPlex did its job and mailed local root, where nothing reads. That is the same shape
+as D1 — a correct detection with a delivery path to nowhere — one layer outside the product. Tracked
+separately as DooPlex hygiene.

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase0-preexisting-scratch/offsite-restore-scratch.listing.txt

@@ -0,0 +1,42 @@
+drwxr-xr-x root/root         0 2026-08-21 18:33 offsite-restore/
+drwxr-xr-x root/root         0 2026-08-21 18:33 offsite-restore/opengist/
+drwxr-xr-x root/root         0 2026-08-03 08:11 offsite-restore/opengist/mnt/
+drwxr-xr-x root/root         0 2026-08-04 14:51 offsite-restore/opengist/mnt/sys_drive/
+drwxr-xr-x root/root         0 2026-08-03 08:29 offsite-restore/opengist/mnt/sys_drive/felhom-data/
+drwxr-xr-x root/root         0 2026-08-04 23:13 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/
+drwxr-xr-x root/root         0 2026-08-06 22:02 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/
+drwxr-xr-x root/root         0 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/
+drwxr-xr-x root/root         0 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/
+-rw-r--r-- root/root      1750 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/.felhom.yml
+-rw-r--r-- root/root      1260 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/docker-compose.yml
+-rw------- root/root       287 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/app.yaml
+drwxr-xr-x root/root         0 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/volume-dumps/
+-rw-r--r-- root/root    182272 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/volume-dumps/opengist_opengist_data.tar
+-rw-r--r-- root/root      1121 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/manifest.json
+drwxr-xr-x root/root         0 2026-08-21 18:30 offsite-restore/calibre-web/
+drwxr-xr-x root/root         0 2026-08-03 08:11 offsite-restore/calibre-web/mnt/
+drwxr-xr-x root/root         0 2026-08-04 14:51 offsite-restore/calibre-web/mnt/sys_drive/
+drwxr-xr-x root/root         0 2026-08-03 08:29 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/
+drwxrwsr-x root/1000         0 2026-08-04 18:45 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/
+drwxr-sr-x root/1000         0 2026-08-04 18:50 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/
+drwxrwsr-x 1000/1000         0 2026-08-09 10:15 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/
+-rw-r--r-- 1000/1000       181 2026-08-04 14:53 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
+drwxrwxr-x 1000/1000         0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/
+-rw-rw-r-- 1000/1000        21 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/plain.txt
+-rw-rw-r-- 1000/1000   3145728 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/binary-3mb.bin
+drwxrwxr-x 1000/1000         0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/nested/
+-rw-rw-r-- 1000/1000        25 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/nested/\305\221szibarack.md
+-rw-rw-r-- 1000/1000        59 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/\303\241rv\303\255zt\305\261r\305\221-t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+-rw-r--r-- 1000/1000     32768 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db-shm
+-rw-r--r-- 1000/1000         0 2026-08-09 04:15 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db-wal
+-rw-r--r-- 1000/1000    413696 2026-08-04 14:52 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db
+drwxr-xr-x root/root         0 2026-08-04 23:13 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/
+drwxr-xr-x root/root         0 2026-08-06 22:02 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/
+drwxr-xr-x root/root         0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/
+drwxr-xr-x root/root         0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/
+-rw-r--r-- root/root      3035 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/.felhom.yml
+-rw-r--r-- root/root      2122 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/docker-compose.yml
+-rw------- root/root       317 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/app.yaml
+drwxr-xr-x root/root         0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/volume-dumps/
+-rw-r--r-- root/root    368640 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/volume-dumps/calibre-web_calibre_web_config.tar
+-rw-r--r-- root/root      1157 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/manifest.json

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase0-preexisting-scratch/offsite-restore-scratch.sha256

@@ -0,0 +1 @@
+c9498bfba3dab7b8c59196be8a6c792f0044356d969b2d81ce4b00b61950aa96  documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase0-preexisting-scratch/offsite-restore-scratch.tar

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experiment/RESULT-MATRIX.txt

@@ -0,0 +1,22 @@
+DRILL 2026-08-21 — FULL/EMPTY contrast on demo-hp. All hashes sha256, byte-for-byte.
+Comparator positive control: one byte flipped at offset 500000 of binary-1mb.bin
+(af -> 00) => sha256sum -c FAILED rc=1; original PASSED rc=0. Mutant discarded.
+
+Planted fixture (5 files, incl. two UTF-8 Hungarian accented names):
+  SENTINEL.txt                    0c23c8531214fe20cbc1ed177da22f51d65570c5aaedb39e380aa8d4d3e62991
+  binary-1mb.bin                  725763bbe679b22d5c231a14083ff13155f475c028c22ade4623955b50a2a84a
+  nested/őszibarack.md            a39ad6f623da67ac72f2e62a24245eef46c722c279ae89cd6b0c7c45afa6a4c4
+  plain.txt                       07e91a985809fc96752f97cddcf6523b211bc607c81a5808befabe35573926f1
+  árvíztűrő-tükörfúrógép.txt      0d6a22ec56acf61b8a80e73eb12cb223d607acaa3684acfcfa7c3e34ecdfabea
+Name bytes (UTF-8 NFC):
+  árvíztűrő-tükörfúrógép.txt = c3a1 72 76 c3ad 7a 74 c5b1 72 c591 2d 74 c3bc 6b c3b6 72 66 c3ba 72 c3b3 67 c3a9 70 2e747874
+  őszibarack.md              = c591 73 7a 69 62 61 72 61 63 6b 2e 6d 64
+
+                     | class      | data leg          | in unit | in offsite snap | checking folder | OFF-SITE restore | LOCAL restore
+ calibre-web  FULL   | drive      | userdata files    |   n/a   | YES 5/5 ident.  | YES 5/5 ident.  | YES 5/5 ident.   | -
+ calibre-web  FULL   | drive      | named volume      | YES 1.42MB | YES          | YES             | NO  (silent)     | -
+ privatebin   FULL   | no-drive   | named volume      | YES 1.06MB | YES 5/5 ident.| YES 5/5 ident.  | REFUSED (false)  | YES 5/5 ident.
+ opengist     EMPTY  | no-drive   | named volume      | YES 181KB skeleton | YES  | YES             | REFUSED (false)  | -
+
+CONCLUSION: the unit and the off-site snapshot HOLD the data, verified by identity.
+The off-site full restore has no named-volume leg at all; the local restore has one.

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experiment/checking-folders-after-full-restore.listing.txt

@@ -0,0 +1,88 @@
+drwxr-xr-x root/root         0 2026-08-21 22:19 offsite-restore/
+drwxr-xr-x root/root         0 2026-08-21 18:33 offsite-restore/opengist/
+drwxr-xr-x root/root         0 2026-08-21 18:00 offsite-restore/opengist/mnt/
+drwxr-xr-x root/root         0 2026-08-21 18:01 offsite-restore/opengist/mnt/sys_drive/
+drwxr-xr-x root/root         0 2026-08-21 18:31 offsite-restore/opengist/mnt/sys_drive/felhom-data/
+drwxr-xr-x root/root         0 2026-08-21 18:31 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/
+drwxr-xr-x root/root         0 2026-08-21 18:31 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/
+drwxr-xr-x root/root         0 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/
+drwxr-xr-x root/root         0 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/
+-rw-r--r-- root/root      1750 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/.felhom.yml
+-rw-r--r-- root/root      1260 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/docker-compose.yml
+-rw------- root/root       287 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/app.yaml
+drwxr-xr-x root/root         0 2026-08-21 22:16 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/volume-dumps/
+-rw-r--r-- root/root    181248 2026-08-21 22:16 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/volume-dumps/opengist_opengist_data.tar
+-rw-r--r-- root/root      1121 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/manifest.json
+drwxr-xr-x root/root         0 2026-08-21 22:19 offsite-restore/privatebin/
+drwxr-xr-x root/root         0 2026-08-21 18:00 offsite-restore/privatebin/mnt/
+drwxr-xr-x root/root         0 2026-08-21 18:01 offsite-restore/privatebin/mnt/sys_drive/
+drwxr-xr-x root/root         0 2026-08-21 18:31 offsite-restore/privatebin/mnt/sys_drive/felhom-data/
+drwxr-xr-x root/root         0 2026-08-21 18:31 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/
+drwxr-xr-x root/root         0 2026-08-21 18:31 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/
+drwxr-xr-x root/root         0 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/
+drwxr-xr-x root/root         0 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/
+-rw-r--r-- root/root      1723 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/.felhom.yml
+-rw-r--r-- root/root      1223 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/docker-compose.yml
+-rw------- root/root       288 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/app.yaml
+drwxr-xr-x root/root         0 2026-08-21 22:16 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/volume-dumps/
+-rw-r--r-- root/root   1055744 2026-08-21 22:16 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/volume-dumps/privatebin_privatebin_data.tar
+-rw-r--r-- root/root      1118 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/manifest.json
+drwxr-xr-x root/root         0 2026-08-21 18:30 offsite-restore/calibre-web/
+drwxr-xr-x root/root         0 2026-08-21 18:00 offsite-restore/calibre-web/mnt/
+drwxr-xr-x nobody/nogroup    0 2026-08-21 18:26 offsite-restore/calibre-web/mnt/felhom-drives/
+drwxr-xr-x root/root         0 2026-07-22 03:30 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/
+drwxrwsr-x root/1000         0 2026-07-21 19:08 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/
+drwxrwsr-x root/1000         0 2026-07-26 08:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/
+drwxrwsr-x 1000/1000         0 2026-08-21 22:10 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/
+-rw-r--r-- 1000/1000       181 2026-08-04 14:53 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-SENTINEL.txt
+drwxrwxr-x 1000/1000         0 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/
+-rw-rw-r-- 1000/1000        35 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/plain.txt
+-rw-rw-r-- 1000/1000   1048576 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/binary-1mb.bin
+drwxrwxr-x 1000/1000         0 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/nested/
+-rw-rw-r-- 1000/1000        21 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/nested/\305\221szibarack.md
+-rw-rw-r-- 1000/1000        46 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/SENTINEL.txt
+-rw-rw-r-- 1000/1000        52 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/\303\241rv\303\255zt\305\261r\305\221-t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+drwxrwxr-x 1000/1000         0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/
+-rw-rw-r-- 1000/1000        21 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/plain.txt
+-rw-rw-r-- 1000/1000   3145728 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/binary-3mb.bin
+drwxrwxr-x 1000/1000         0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/nested/
+-rw-rw-r-- 1000/1000        25 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/nested/\305\221szibarack.md
+-rw-rw-r-- 1000/1000        59 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/\303\241rv\303\255zt\305\261r\305\221-t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+-rw-r--r-- 1000/1000     32768 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/metadata.db-shm
+-rw-r--r-- 1000/1000         0 2026-08-09 04:15 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/metadata.db-wal
+-rw-r--r-- 1000/1000    413696 2026-08-04 14:52 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/metadata.db
+drwxr-xr-x root/root         0 2026-07-23 12:10 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/
+drwxr-xr-x root/root         0 2026-08-21 18:31 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/
+drwxr-xr-x root/root         0 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/
+drwxr-xr-x root/root         0 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/
+-rw-r--r-- root/root      3035 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/.felhom.yml
+-rw-r--r-- root/root      2122 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/docker-compose.yml
+-rw------- root/root       327 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/app.yaml
+drwxr-xr-x root/root         0 2026-08-21 22:16 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/volume-dumps/
+-rw-r--r-- root/root   1422848 2026-08-21 22:16 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/volume-dumps/calibre-web_calibre_web_config.tar
+-rw-r--r-- root/root      1165 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/manifest.json
+drwxr-xr-x root/root         0 2026-08-04 14:51 offsite-restore/calibre-web/mnt/sys_drive/
+drwxr-xr-x root/root         0 2026-08-03 08:29 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/
+drwxrwsr-x root/1000         0 2026-08-04 18:45 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/
+drwxr-sr-x root/1000         0 2026-08-04 18:50 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/
+drwxrwsr-x 1000/1000         0 2026-08-09 10:15 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/
+-rw-r--r-- 1000/1000       181 2026-08-04 14:53 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
+drwxrwxr-x 1000/1000         0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/
+-rw-rw-r-- 1000/1000        21 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/plain.txt
+-rw-rw-r-- 1000/1000   3145728 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/binary-3mb.bin
+drwxrwxr-x 1000/1000         0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/nested/
+-rw-rw-r-- 1000/1000        25 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/nested/\305\221szibarack.md
+-rw-rw-r-- 1000/1000        59 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/\303\241rv\303\255zt\305\261r\305\221-t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+-rw-r--r-- 1000/1000     32768 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db-shm
+-rw-r--r-- 1000/1000         0 2026-08-09 04:15 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db-wal
+-rw-r--r-- 1000/1000    413696 2026-08-04 14:52 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db
+drwxr-xr-x root/root         0 2026-08-04 23:13 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/
+drwxr-xr-x root/root         0 2026-08-06 22:02 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/
+drwxr-xr-x root/root         0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/
+drwxr-xr-x root/root         0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/
+-rw-r--r-- root/root      3035 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/.felhom.yml
+-rw-r--r-- root/root      2122 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/docker-compose.yml
+-rw------- root/root       317 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/app.yaml
+drwxr-xr-x root/root         0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/volume-dumps/
+-rw-r--r-- root/root    368640 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/volume-dumps/calibre-web_calibre_web_config.tar
+-rw-r--r-- root/root      1157 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/manifest.json

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experiment/checking-folders-after-full-restore.sha256

@@ -0,0 +1 @@
+7e59e57d6458d28f530dbaddbee0f2314ea1ef885052701531f57bad2529849d  documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experiment/checking-folders-after-full-restore.tar

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experiment/messages-verbatim.txt

@@ -0,0 +1,32 @@
+VERBATIM customer-facing outcome strings, read from /api/backup/restore-status
+(the same value the wizard banner renders). Times CEST.
+
+1) 22:21:51  off-site FULL RESTORE (reconstitute), app = privatebin  [40-class, no HDD_PATH]
+   ok = FALSE
+   "A teljes visszaállítás sikertelen: a(z) privatebin nincs telepítve, ezért nincs hová
+    visszaállítani az adatait. A mentése szerint az adatai itt voltak: /mnt/sys_drive.
+    Telepítsd újra az alkalmazást (Alkalmazások) ugyanerre a helyre, utána ez a
+    visszaállítás működni fog"
+   FACT AT THAT MOMENT: privatebin deployed=true, state=running, container healthy.
+
+2) 22:23:36  off-site FULL RESTORE (reconstitute), app = calibre-web  [drive class]
+   ok = TRUE
+   "A(z) calibre-web: 5 fájl visszaállítva (mentés: 2026-08-21 22:16) — az alkalmazás
+    újraindult. Ennek az alkalmazásnak nincs adatbázisa."
+   FACT: the 5 declared-userdata files came back byte-identical.
+         The named volume calibre-web_calibre_web_config was NOT restored, though its
+         1,422,848-byte tar was in the snapshot, in the checking folder, and named in
+         manifest.json volume_dumps. The message does not mention it.
+
+3) 22:25:31  LOCAL restore from recovery unit, app = privatebin
+   ok = TRUE
+   "privatebin visszaállítva (helyi)."
+   FACT: the named volume WAS restored, all 5 planted files byte-identical.
+         The message carries no counts at all - it reads the same whatever happened.
+
+4) 22:13:15  off-site backup run with ZERO apps selected
+   log:  "[offbox] backup run started (0 app(s) toggled)"
+         "[offbox] backup OK: 0 app(s) backed up, 18 snapshot(s), 14s"
+   card: badge "Aktív — nincs kijelölt alkalmazás" / "✓ Rendben" /
+         "Sikeres — nincs mentésre jelölt alkalmazás" /
+         "Nincs távoli mentésre jelölt alkalmazás — jelölj ki legalább egyet."

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experiment/plant-manifest.txt

@@ -0,0 +1,5 @@
+0c23c8531214fe20cbc1ed177da22f51d65570c5aaedb39e380aa8d4d3e62991  DRILL-2026-08-21/SENTINEL.txt
+725763bbe679b22d5c231a14083ff13155f475c028c22ade4623955b50a2a84a  DRILL-2026-08-21/binary-1mb.bin
+a39ad6f623da67ac72f2e62a24245eef46c722c279ae89cd6b0c7c45afa6a4c4  DRILL-2026-08-21/nested/őszibarack.md
+07e91a985809fc96752f97cddcf6523b211bc607c81a5808befabe35573926f1  DRILL-2026-08-21/plain.txt
+0d6a22ec56acf61b8a80e73eb12cb223d607acaa3684acfcfa7c3e34ecdfabea  DRILL-2026-08-21/árvíztűrő-tükörfúrógép.txt

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase4-abandon/repo-inventory-before.txt

@@ -0,0 +1,59 @@
+ID        Time                 Host        Tags                         Paths
+------------------------------------------------------------------------------------------------------------------------------
+e6132ae5  2026-08-04 19:36:26  demo-hp     felhom-offbox,calibre-web    /mnt/sys_drive/felhom-data/backups/primary/calibre-web
+                                                                        /mnt/sys_drive/felhom-data/userdata/media/books
+
+9ac78c98  2026-08-04 19:36:31  demo-hp     felhom-offbox,opengist       /mnt/sys_drive/felhom-data/backups/primary/opengist
+
+92212ff8  2026-08-04 19:36:36  demo-hp     felhom-offbox,privatebin     /mnt/sys_drive/felhom-data/backups/primary/privatebin
+
+9d6d233f  2026-08-05 09:12:42  demo-hp     felhom-offbox,calibre-web    /mnt/sys_drive/felhom-data/backups/primary/calibre-web
+                                                                        /mnt/sys_drive/felhom-data/userdata/media/books
+
+29e7b245  2026-08-05 09:12:48  demo-hp     felhom-offbox,opengist       /mnt/sys_drive/felhom-data/backups/primary/opengist
+
+cd1db049  2026-08-05 09:12:53  demo-hp     felhom-offbox,privatebin     /mnt/sys_drive/felhom-data/backups/primary/privatebin
+
+0ef7a006  2026-08-06 20:00:44  demo-hp     felhom-offbox,calibre-web    /mnt/sys_drive/felhom-data/backups/primary/calibre-web
+                                                                        /mnt/sys_drive/felhom-data/userdata/media/books
+
+8662a8c1  2026-08-06 20:00:50  demo-hp     felhom-offbox,opengist       /mnt/sys_drive/felhom-data/backups/primary/opengist
+
+6dfa6602  2026-08-06 20:00:54  demo-hp     felhom-offbox,privatebin     /mnt/sys_drive/felhom-data/backups/primary/privatebin
+
+3635f945  2026-08-07 02:15:13  demo-hp     felhom-offbox,calibre-web    /mnt/sys_drive/felhom-data/backups/primary/calibre-web
+                                                                        /mnt/sys_drive/felhom-data/userdata/media/books
+
+7b1fa8b0  2026-08-07 02:15:17  demo-hp     felhom-offbox,opengist       /mnt/sys_drive/felhom-data/backups/primary/opengist
+
+38edf5b8  2026-08-07 02:15:21  demo-hp     felhom-offbox,privatebin     /mnt/sys_drive/felhom-data/backups/primary/privatebin
+
+a4d03ee3  2026-08-08 02:15:12  demo-hp     felhom-offbox,calibre-web    /mnt/sys_drive/felhom-data/backups/primary/calibre-web
+                                                                        /mnt/sys_drive/felhom-data/userdata/media/books
+
+f534bffe  2026-08-08 02:15:17  demo-hp     felhom-offbox,opengist       /mnt/sys_drive/felhom-data/backups/primary/opengist
+
+a685d30e  2026-08-08 02:15:22  demo-hp     felhom-offbox,privatebin     /mnt/sys_drive/felhom-data/backups/primary/privatebin
+
+41c830db  2026-08-09 08:30:38  demo-hp     felhom-offbox,calibre-web    /mnt/sys_drive/felhom-data/backups/primary/calibre-web
+                                                                        /mnt/sys_drive/felhom-data/userdata/media/books
+
+9e38b84c  2026-08-09 08:30:49  demo-hp     felhom-offbox,opengist       /mnt/sys_drive/felhom-data/backups/primary/opengist
+
+78b93f04  2026-08-09 08:30:53  demo-hp     felhom-offbox,privatebin     /mnt/sys_drive/felhom-data/backups/primary/privatebin
+
+8c44bd4c  2026-08-21 21:00:52  demo-hp     felhom-offbox,calibre-web    /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
+                                                                        /mnt/felhom-drives/hdd_1/userdata/media/books
+
+07bac5ad  2026-08-21 21:00:55  demo-hp     felhom-offbox,privatebin     /mnt/sys_drive/felhom-data/backups/primary/privatebin
+
+7e4b703b  2026-08-21 21:00:58  demo-hp     felhom-offbox,kimai          /mnt/sys_drive/felhom-data/backups/primary/kimai
+
+16cb8ce7  2026-08-21 21:01:08  demo-hp     felhom-offbox,paperless-ngx  /mnt/felhom-drives/hdd_1/appdata/paperless/media
+                                                                        /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
+
+2a891149  2026-08-21 21:01:13  demo-hp     felhom-offbox,romm           /mnt/felhom-drives/hdd_1/backups/primary/romm
+
+49b9f317  2026-08-21 21:01:22  demo-hp     felhom-offbox,opengist       /mnt/sys_drive/felhom-data/backups/primary/opengist
+------------------------------------------------------------------------------------------------------------------------------
+24 snapshots

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase4-abandon/sweep-result.txt

@@ -0,0 +1,23 @@
+PART 4.2 — the abandonment sweep, watched firing. 2026-08-22.
+
+State created 2026-08-21 23:08-23:09 CEST:
+  set-aside store  u629488-sub3:/home/felhom-repo-superseded-drill-20260821  (config/data/index/snapshots)
+  countdown        started 2026-08-07, due 2026-08-20  (written into settings.json, controller restarted)
+  product CLI      "abandonment countdown RUNNING … deleted on: 2026-08-20 … days left: 0"
+
+05:10 CEST — the daily `offsite-abandon-sweep` job fired.
+  /home on the storage box now stamped 03:10Z
+  /home/felhom-repo-superseded-drill-20260821   -> GONE
+  /home/felhom-repo  (the LIVE repository)      -> present, mtime still Aug 4  ** survived **
+
+05:13 CEST — the hub half. Event 3025 `offsite_abandon_purged`:
+  "Az ügyfél korábbi távoli mentései és a hozzájuk tartozó megőrzött helyreállítási csomag is
+   törölve (1 csomag). Az ügyfél döntése alapján, a 14 napos türelmi idő lejárta után."
+  demo-hp's row in host_escrow_superseded: REMOVED (there was exactly one).
+  demo-felhom's rows 4, 11, 12: ALL PRESENT — the protected fixtures were not touched.
+
+Controller closed itself out: every abandon_* field removed from settings.json;
+`--abandon-status` reports "no abandonment countdown is running on this box".
+
+VERDICT: the two-phase commit's promise — "it removes BOTH halves or neither" — is confirmed
+live for the first time. It removed exactly the recorded set-aside path and nothing else.

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperless/FINDING.txt

@@ -0,0 +1,34 @@
+paperless-ngx: the database is dumped into a directory for a stack that does not exist,
+so the recovery unit never contains it, and the restore then tells the customer the app
+has no database. Proven live 2026-08-21 22:39-22:45 CEST on demo-hp.
+
+MECHANISM (source):
+  internal/appbackup/dbdump.go:770 deriveStackName("paperless-postgres", known)
+    1. candidate = suffixStripStackName("paperless-postgres") = "paperless"   (line 801)
+    2. known is non-empty, known["paperless"] is FALSE   (the stack is "paperless-ngx")
+    3. known["paperless-postgres"] is FALSE
+    4. no known stack name is a prefix of "paperless-postgres"  ("paperless-ngx" is not)
+    5. FALLS THROUGH to `return candidate`  (line 797) -> "paperless"
+  An unresolved mapping is returned as if resolved. There is no warning and no refusal.
+
+OBSERVED CONSEQUENCES (all live):
+  a) the dump is written to
+       /mnt/sys_drive/felhom-data/backups/primary/paperless/db-dumps/paperless-postgres.sql
+     284,617 bytes, 72 tables, valid=true  -- an orphan directory for a non-existent stack,
+     on the SYSTEM drive, while the app's real unit is on /mnt/felhom-drives/hdd_1.
+  b) the real unit /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx/manifest.json
+     records  "db_dumps": null
+  c) the off-site snapshot therefore carries no .sql at all
+     (checking folder: `find ... -name "*.sql" | wc -l` = 0)
+  d) writeSafetyDump (offbox_reconstitute.go:115) filters discovered DBs on
+     db.StackName == stackName, so `mine` is empty -> returns ("", nil) -> hasDB = false.
+     NO pre-restore safety dump is taken.  Verified: `find /mnt -name "pre-restore-*"`
+     returned nothing before AND after the destructive restore.
+  e) the destructive restore ran to completion and reported SUCCESS:
+       "A(z) paperless-ngx: 0 fájl visszaállítva (mentés: 2026-08-21 22:41)
+        — az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa."
+     The controller had dumped that same database 5 minutes earlier.
+
+The orphan directory is also invisible to the app's off-site push, because the push
+resolves paths from the app's own unit path -- so the only copy of that database dump
+is on the system drive of the machine it protects.

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperless/discovery-log.txt

@@ -0,0 +1,9 @@
+{"timestamp":"2026-08-21T20:38:46Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}
+{"timestamp":"2026-08-21T20:39:37Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}
+{"timestamp":"2026-08-21T20:39:37Z","level":"DEBUG","message":"DumpOne: starting dump for container=paperless-postgres, stack=paperless, dbType=postgres, dumpDir=/mnt/sys_drive/felhom-data/backups/primary/paperless/db-dumps","source":"dbdump.go:189"}
+{"timestamp":"2026-08-21T20:39:37Z","level":"DEBUG","message":"DumpOne: completed paperless-postgres → paperless-postgres.sql (size=277.9 KB, valid=true, tables=72, duration=313ms)","source":"dbdump.go:330"}
+{"timestamp":"2026-08-21T20:41:23Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}
+{"timestamp":"2026-08-21T20:41:23Z","level":"DEBUG","message":"DumpOne: starting dump for container=paperless-postgres, stack=paperless, dbType=postgres, dumpDir=/mnt/sys_drive/felhom-data/backups/primary/paperless/db-dumps","source":"dbdump.go:189"}
+{"timestamp":"2026-08-21T20:41:23Z","level":"DEBUG","message":"DumpOne: completed paperless-postgres → paperless-postgres.sql (size=278.2 KB, valid=true, tables=72, duration=314ms)","source":"dbdump.go:330"}
+{"timestamp":"2026-08-21T20:43:46Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}
+{"timestamp":"2026-08-21T20:44:33Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperless/orphan-dump-and-manifest.txt

@@ -0,0 +1,48 @@
+total 288
+drwxr-xr-x 2 root root   4096 Aug 21 20:41 .
+drwxr-xr-x 3 root root   4096 Aug 21 20:39 ..
+-rw-r--r-- 1 root root 284902 Aug 21 20:41 paperless-postgres.sql
+--- real unit:
+{
+  "schema_version": 2,
+  "app_name": "paperless-ngx",
+  "display_name": "Paperless-ngx",
+  "controller_version": "0.217.0",
+  "created_at": "2026-08-21T20:42:01Z",
+  "drive": "/mnt/felhom-drives/hdd_1",
+  "namespace_root": "/mnt/felhom-drives/hdd_1",
+  "image_pins": [
+    "ghcr.io/paperless-ngx/paperless-ngx:2.20.15",
+    "postgres:16-alpine",
+    "redis:7-alpine"
+  ],
+  "secret_env_vars": [
+    "DB_PASSWORD",
+    "PAPERLESS_SECRET_KEY",
+    "PAPERLESS_ADMIN_PASSWORD"
+  ],
+  "data_key_env_vars": null,
+  "secret_source": "portable secrets (data keys, DB passwords, internal signing secrets) are IN this unit's compose/app.yaml (0600); internet-reachable admin logins are NOT, and come from the guest's app.yaml or are regenerated on restore",
+  "config_files": [
+    "docker-compose.yml",
+    ".felhom.yml",
+    "app.yaml"
+  ],
+  "db_dumps": null,
+  "volume_dumps": [
+    "paperless-ngx_paperless_data.tar",
+    "paperless-ngx_paperless_postgres_data.tar",
+    "paperless-ngx_paperless_redis_data.tar"
+  ],
+  "checksums": {
+    ".felhom.yml": "a7cce0a557fd151e6385a137f4721366dd2cd0aa3876783f0f9f0acc9a78dd23",
+    "app.yaml": "8dfec452dfc00c7e3dce26864bf97165acac44f32470d2425c96e2bd86649cf0",
+    "docker-compose.yml": "b112952565ef3928f192358ea58fdf4a5a26d788593bf2614e36050c77539b3d"
+  },
+  "portable_secret_env_vars": [
+    "DB_PASSWORD",
+    "PAPERLESS_SECRET_KEY"
+  ],
+  "offsite_run_id": "20260821T204123Z",
+  "dumps_at": "2026-08-21T20:41:23Z"
+}

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase4-part4/results.txt

@@ -0,0 +1,64 @@
+PART 4 — things we claim and have never watched. demo-hp, 2026-08-21, times CEST.
+
+4.1 THE DESTRUCTIVE RESTORE
+  (a) "nothing is ever deleted"  -- PASS, both directions, calibre-web 22:26:59.
+      POST-SNAPSHOT.txt, created after the snapshot, SURVIVED the restore.
+      plain.txt, mutated after the snapshot, was OVERWRITTEN back to the snapshot's
+      content (sha 07e91a98…). Copier is rsync -a, no --delete, no --ignore-existing
+      (offbox_reconstitute.go:94).
+      NOTE ON THE COUNT: the message says "2 fájl visszaállítva" because rsync counts
+      TRANSFERS, not files restored. An identical restore reports "0 fájl visszaállítva",
+      which is indistinguishable from a restore that did nothing.
+
+  (b) "a safety dump is taken and VERIFIED before anything is stopped, and the whole
+      operation refuses if it cannot be"
+      HAPPY PATH -- PASS, romm 23:02:44.
+        21:02:47Z  "[offbox] romm: pre-restore safety dump written →
+                    pre-restore-20260821T210246Z-romm-mariadb.sql (60.8 KB)"
+        21:02:47Z  "[stacks] StopStack romm: current state=running"
+        The dump precedes the stop. Message correctly said
+        "0 fájl és az adatbázis visszaállítva".
+      THE REFUSAL -- PASS, romm 23:04:15.
+        Safety dump made impossible by putting a regular FILE at the db-dumps path.
+        Result: ok=FALSE,
+          "A teljes visszaállítás sikertelen: a biztonsági mentés könyvtára nem hozható
+           létre: mkdir /mnt/felhom-drives/hdd_1/backups/primary/romm/db-dumps:
+           not a directory"
+        Nothing changed: plain.txt kept my post-snapshot mutation (sha 3754dfc6…), and
+        romm's container StartedAt was unchanged (21:03:08) -- the app was never stopped.
+        JUDGEMENT: honest and it names the path, but it leaks a raw Go mkdir error into
+        a customer surface.
+      THE HOLE -- the guard only protects apps whose database the discovery resolves to
+        the right stack. For paperless-ngx it concludes "no database", so hasDB is false
+        and the refusal CANNOT fire: the undo is absent rather than refused. See
+        ../phase4-paperless/FINDING.txt.
+      SIDE EFFECT, NOT PREVIOUSLY FILED -- the safety dump DESTROYS the unit's own DB dump.
+        DumpOne writes the canonical `<stack>-<dbtype>.sql` (appbackup/dbdump.go:200-202),
+        i.e. the app's real dump, and only THEN is it renamed to pre-restore-*.
+        The comment at offbox_reconstitute.go:147-148 states it "can never overwrite the
+        app's real dump". It does.
+        PROVEN: romm's db-dumps held romm-mariadb.sql (62,270 B) at 22:59; after one
+        reconstitute it held ONLY pre-restore-20260821T210246Z-romm-mariadb.sql.
+        Consequence: until the next backup run the LOCAL restore-from-unit finds no .sql
+        and tells the customer the app has no database.
+
+4.3 A DAMAGED STORE  -- MIXED
+  Method: one byte flipped inside pack 967853d2… (offset 5,000,000) via the repo's own
+  SFTP transport; the pack's name is its content hash, so this is genuine corruption.
+  * `restic check` DOES detect it ("ciphertext verification failed",
+    "Fatal: repository contains errors").  BUT the controller NEVER RUNS `restic check`:
+    the only restic verbs in the whole controller are restore, snapshots, backup, unlock,
+    stats, init, forget, prune, cat. The agent's restore-test is PBS-tier only.
+    So the off-site store is never verified by any layer, at any time.
+  * A restore that TOUCHES the damage fails honestly:
+    ok=FALSE, "A visszaállítás sikertelen: offbox restore paperless-ngx: exit status 1:
+    … ignoring error for …/documents/originals/0000011.pdf: ciphertext verification failed"
+  * BUT the failure is not remembered. It left a PARTIAL scratch (78 MB, 54 files,
+    15 of 16 originals). OffboxFullScratchReady (offbox_restore.go:305) only asks
+    "does the directory exist and is it non-empty", so the wizard then offered all three
+    actions including "Teljes visszaállítás indítása".
+  * Pressing it ran the DESTRUCTIVE restore from that known-incomplete copy and reported
+    SUCCESS: ok=TRUE, "A(z) paperless-ngx: 0 fájl visszaállítva … Ennek az alkalmazásnak
+    nincs adatbázisa."
+  REPO REPAIRED afterwards from the byte-identical originals; `restic check` now says
+  "no errors were found".

documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase5-scheduled/scheduled-cycle-and-sweep.log

@@ -0,0 +1,71 @@
+[2026-08-21 23:34:08 CEST] collector v2 started (epoch waits)
+[2026-08-22 02:38:09 CEST] === after the 02:30 SCHEDULED local cycle
+[2026-08-22 02:38:09 CEST] --- units:
+## /mnt/sys_drive/felhom-data/backups/primary/kimai
+        2235  2026-08-21 21:00  compose/.felhom.yml
+         488  2026-08-21 21:00  compose/app.yaml
+        2195  2026-08-21 21:00  compose/docker-compose.yml
+       48217  2026-08-22 00:30  db-dumps/kimai-mariadb.sql
+        1275  2026-08-21 21:00  manifest.json
+   160331776  2026-08-22 00:30  volume-dumps/kimai_kimai_db_data.tar
+    52845056  2026-08-22 00:30  volume-dumps/kimai_kimai_var.tar
+## /mnt/sys_drive/felhom-data/backups/primary/opengist
+        1750  2026-08-21 21:00  compose/.felhom.yml
+         287  2026-08-21 21:00  compose/app.yaml
+        1260  2026-08-21 21:00  compose/docker-compose.yml
+        1121  2026-08-21 21:00  manifest.json
+      181248  2026-08-22 00:30  volume-dumps/opengist_opengist_data.tar
+## /mnt/sys_drive/felhom-data/backups/primary/paperless
+      294936  2026-08-22 00:30  db-dumps/paperless-postgres.sql
+## /mnt/sys_drive/felhom-data/backups/primary/privatebin
+        1723  2026-08-21 21:00  compose/.felhom.yml
+         288  2026-08-21 21:00  compose/app.yaml
+        1223  2026-08-21 21:00  compose/docker-compose.yml
+        1118  2026-08-21 21:00  manifest.json
+     1055744  2026-08-22 00:30  volume-dumps/privatebin_privatebin_data.tar
+## /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
+        3035  2026-08-21 21:00  compose/.felhom.yml
+         327  2026-08-21 21:00  compose/app.yaml
+        2122  2026-08-21 21:00  compose/docker-compose.yml
+        1165  2026-08-21 21:00  manifest.json
+      368640  2026-08-22 00:30  volume-dumps/calibre-web_calibre_web_config.tar
+## /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
+        5971  2026-08-21 21:00  compose/.felhom.yml
+         664  2026-08-21 21:00  compose/app.yaml
+        5802  2026-08-21 21:00  compose/docker-compose.yml
+        1462  2026-08-21 21:00  manifest.json
+      231424  2026-08-22 00:30  volume-dumps/paperless-ngx_paperless_data.tar
+    71417344  2026-08-22 00:30  volume-dumps/paperless-ngx_paperless_postgres_data.tar
+      116736  2026-08-22 00:30  volume-dumps/paperless-ngx_paperless_redis_data.tar
+## /mnt/felhom-drives/hdd_1/backups/primary/romm
+        5520  2026-08-21 21:20  compose/.felhom.yml
+         576  2026-08-21 21:20  compose/app.yaml
+        4399  2026-08-21 21:20  compose/docker-compose.yml
+       62270  2026-08-21 21:02  db-dumps/pre-restore-20260821T210246Z-romm-mariadb.sql
+       62270  2026-08-22 00:30  db-dumps/romm-mariadb.sql
+        1466  2026-08-21 21:20  manifest.json
+        2560  2026-08-22 00:30  volume-dumps/romm_romm_config.tar
+   160247296  2026-08-22 00:30  volume-dumps/romm_romm_db_data.tar
+    14677504  2026-08-22 00:30  volume-dumps/romm_romm_redis_data.tar
+[2026-08-22 02:38:10 CEST] --- planted files still byte-identical (calibre-web books):
+eee5880e304b27c13d205aac9989e901a5f2393be7ecf076ee2c3a0195a2358c  DRILL-2026-08-21/POST-SNAPSHOT.txt
+0c23c8531214fe20cbc1ed177da22f51d65570c5aaedb39e380aa8d4d3e62991  DRILL-2026-08-21/SENTINEL.txt
+725763bbe679b22d5c231a14083ff13155f475c028c22ade4623955b50a2a84a  DRILL-2026-08-21/binary-1mb.bin
+a39ad6f623da67ac72f2e62a24245eef46c722c279ae89cd6b0c7c45afa6a4c4  DRILL-2026-08-21/nested/őszibarack.md
+07e91a985809fc96752f97cddcf6523b211bc607c81a5808befabe35573926f1  DRILL-2026-08-21/plain.txt
+0d6a22ec56acf61b8a80e73eb12cb223d607acaa3684acfcfa7c3e34ecdfabea  DRILL-2026-08-21/árvíztűrő-tükörfúrógép.txt
+[2026-08-22 02:38:12 CEST] --- scheduled db-dump log:
+grep: /opt/docker/felhom-controller/data/debug-ring.log: No such file or directory
+[2026-08-22 04:30:13 CEST] === after the 04:15 SCHEDULED off-site run
+{"last_duration":"2m8s","last_error":"","last_run":"2026-08-22T02:17:12Z","orphaned":false,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":0,"current_file":"","elapsed_sec":0,"phase":""},"repo_size_human":"42.8 MB","snapshots":27,"status":"ok"}
+
+[2026-08-22 04:30:13 CEST] --- offsite log:
+grep: /opt/docker/felhom-controller/data/debug-ring.log: No such file or directory
+[2026-08-22 05:20:15 CEST] === after the 05:10 abandonment sweep
+[2026-08-22 05:20:15 CEST] --- sweep log:
+grep: /opt/docker/felhom-controller/data/debug-ring.log: No such file or directory
+[2026-08-22 05:20:16 CEST] --- product CLI:
+[INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json
+no abandonment countdown is running on this box
+[2026-08-22 05:20:17 CEST] --- settings abandon fields:
+[2026-08-22 05:20:18 CEST] collector finished

documentation/audits/DRILL-cooldown-grain-2026-08-23/README.md

@@ -0,0 +1,118 @@
+# DRILL — R-389: only the first broken app per hour reached the operator (2026-08-23)
+
+**Hub v0.107.0 → v0.108.0. No controller release, so no golden and no floor.** Live leg on `demo-hp`
+(Tier 0) plus the hub. UNATTENDED. Method: the hub's own SQLite records plus the controller log.
+Guest and hub DB are UTC; the hub pod logs CEST.
+
+**No halt condition fired.**
+
+## The pair that says it
+
+Same box, same shape — two apps down four minutes apart, inside one hour:
+
+```
+2026-08-23 09:27:51  sent        BookStack     <- v0.107.0
+2026-08-23 09:31:51  suppressed  PrivateBin        operator cooldown 1h, key=demo-hp:app_start_failed
+
+2026-08-23 11:56:57  sent        OpenGist      <- v0.108.0
+2026-08-23 12:00:57  sent        Calibre-Web
+```
+
+**2 sent / 0 suppressed**, where the day before the identical shape gave one of each.
+
+And the keys, from the hub's own suppression rows — it records the key only when it declines:
+
+```
+operator cooldown 1h, key=demo-hp:app_start_failed:opengist
+operator cooldown 1h, key=demo-hp:app_start_failed:calibre-web
+```
+
+against v0.107.0's shared `key=demo-hp:app_start_failed`.
+
+## The fence, and why it is not decoration
+
+`crossdrive_failed` is severity `error`, reaches the operator leg, and carries `stack_name` — through
+`CrossDriveDetails`, **a different struct from `AppDetails`**. A rule of the form *"if the details
+carry a stack_name, split per app"* would have split it and silently undone R-182.
+
+Proven live: two different apps' `crossdrive_failed`, one minute apart →
+
+```
+sent        opengist
+suppressed  calibre-web    operator cooldown 1h, key=demo-hp:crossdrive_failed
+```
+
+**Byte-identical to the derived v0.107.0 key. No app suffix.** That is why `cooldownStackSuffix`
+takes the event type as well as the details, unlike its two siblings.
+
+## Part 2 — the burst, measured
+
+Three apps stopped in one scan (`kimai`, `romm`, `paperless-ngx`, none with a live cooldown — checked
+first, because a stale one would have halved the count and made the answer look better than it is):
+
+| | |
+|---|---|
+| attempted | **3** |
+| sent | **3** |
+| suppressed | **0** |
+
+**Judgement: per-app is the right grain, and this volume is acceptable.** The reference box has 8
+deployed apps, so a total outage is 8 mails; the boot grace (90 s), the quiesce grace (180 s) and the
+per-app edge trigger absorb reboots, backup cycles and persistently-dead apps. **No digest row was
+filed.** The reopening condition is stated rather than left implicit: it scales linearly with app
+count and has no ceiling, so a box large enough that a total outage is unreadable is the point at
+which the answer becomes a digest with a customer message — not a wider cooldown.
+
+## Part 3 — gate 11, and the spec discrepancy it forced
+
+The gate refuses a push whose `REPORT.md` carries an observation with neither `FILED: R-NNN` nor
+`NOT-A-FINDING: <reason>`.
+
+**The specification said an item may "cite an R-NNN that resolves". That rule would have passed the
+very item the gate was built to catch.** Yesterday's lost observation reads *"This is R-182's known
+cooldown-key shape…"* — `R-182` resolves, and it is cited as an **analogy**, not as the row that files
+it. No parser can tell citation-as-precedent from citation-as-filing by reading prose. The marker is
+therefore explicit, and the discrepancy is recorded in the gate's docstring rather than quietly
+resolved. `EDGE 7` in the evidence is that exact case, convicted.
+
+| Control | Expected | Observed |
+|---|---|---|
+| historical: yesterday's real section, verbatim | refuse | **exit 1**, naming both items |
+| plant an observation with no row | refuse | **exit 1** |
+| add the row | pass | **exit 0** |
+| remove the observation | pass quietly | **exit 0** |
+| `FILED:` a row that does not resolve | refuse | **exit 1** |
+| `NOT-A-FINDING:` with no reason | refuse | **exit 1** |
+| both markers on one item | refuse | **exit 1** |
+| section present, no numbered items | inconclusive | **exit 2**, saying what it could not read |
+| no `REPORT.md` at all | inconclusive | **exit 2** |
+| a bare `R-182` mention (the trap) | refuse | **exit 1** |
+
+## Evidence index (`evidence/`)
+
+| File | What it shows |
+|---|---|
+| `redproof-1-key.txt` | suffix dropped from the key → `1 operator mail(s), want 2`, and the live key shape reproduced |
+| `redproof-2-allowlist.txt` | allow-list removed → `crossdrive_failed`, `app_deployed`, `app_removed`, `backup_failed` all split per app |
+| `gate11-01-historical-redproof.txt` | yesterday's actual observations section, refused |
+| `gate11-02-three-controls.txt` | plant → refuse, file → pass, remove → pass |
+| `gate11-03-edges.txt` | seven boundaries incl. the R-182 trap |
+| `live-01`…`live-05` | Scenarios A and B: both sent, then each suppressed under its own key |
+| `live-06`, `live-07` | Scenario C: crossdrive stays coarse |
+| `live-08`, `live-09` | the burst, with its pre-check |
+| `live-10-full-controller-log.txt` | 1803 lines, pulled before the restore |
+
+## Teardown
+
+Nothing provisioned. Five apps were stopped across the walk (`opengist`, `calibre-web`, `kimai`,
+`romm`, `paperless-ngx`) and **all were restarted and confirmed healthy** — 17 containers up. No app
+was rebuilt, redeployed or restored; the three retained subjects (`docmost`, `bookstack`,
+`privatebin`) were not touched at all.
+
+**Hub-side, stated explicitly.** The hub was written this session: deployment to v0.108.0 via the
+manifest, and **six probe events were POSTed to the live hub** for Scenarios B and C (two
+`app_start_failed`, two `crossdrive_failed`, plus the two from yesterday's Scenario H that were
+deliberately left in place). They are inert event rows for `demo-hp` and are named here rather than
+left to be found. **Every count in this drill is filtered by `created_at >= T0`** precisely so those
+rows cannot contaminate it. Nothing else: no appliance registered, no customer created, no artifact
+manifest changed, floor untouched.

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/gate11-01-historical-redproof.txt

@@ -0,0 +1,16 @@
+### HISTORICAL RED-PROOF — yesterday's ACTUAL observations section, verbatim from 2f7c9a6 ###
+  report            : ../../../../../tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/3b17bdbd-9113-4908-9b30-15678f2978bc/scratchpad/gate11/REPORT.md
+  section           : ## 10. Observations — recorded, not acted on
+  observation items : 2
+
+OBSERVATIONS GATE FAILED: 2 observation(s) with nothing behind them.
+
+  item 1: **The operator cooldown key carries no app identifier.** PrivateBin's alarm fo…
+     neither `FILED: R-NNN` nor `NOT-A-FINDING: <reason>`
+
+  item 2: Two probe events remain as rows for `demo-hp` from Scenario H — inert, and nam…
+     neither `FILED: R-NNN` nor `NOT-A-FINDING: <reason>`
+
+An observation that lives only in REPORT.md has a lifetime of ONE SESSION — this file is overwritten every time. That is how the cooldown-grain finding was lost on 2026-08-23 and had to be re-derived the next day.
+Fix: add `FILED: R-NNN` naming the row you opened for it, or `NOT-A-FINDING: <why this is not worth a row>`. Opening the row is the default; declaring is the exception and needs its reason stated.
+EXIT=1 (want 1)

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/gate11-02-three-controls.txt

@@ -0,0 +1,25 @@
+### CONTROL 1 — an observation with NO row: must REFUSE (1) ###
+  report            : ../../../../../tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/3b17bdbd-9113-4908-9b30-15678f2978bc/scratchpad/gate11/c1.md
+  section           : ## 9. Observations
+  observation items : 1
+
+OBSERVATIONS GATE FAILED: 1 observation(s) with nothing behind them.
+
+  item 1: The operator cooldown key carries no app identifier, so only the first app per…
+     neither `FILED: R-NNN` nor `NOT-A-FINDING: <reason>`
+
+An observation that lives only in REPORT.md has a lifetime of ONE SESSION — this file is overwritten every time. That is how the cooldown-grain finding was lost on 2026-08-23 and had to be re-derived the next day.
+Fix: add `FILED: R-NNN` naming the row you opened for it, or `NOT-A-FINDING: <why this is not worth a row>`. Opening the row is the default; declaring is the exception and needs its reason stated.
+EXIT=1
+
+### CONTROL 2 — add the row: must PASS (0) ###
+  report            : ../../../../../tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/3b17bdbd-9113-4908-9b30-15678f2978bc/scratchpad/gate11/c2.md
+  section           : ## 9. Observations
+  observation items : 1
+      OK  1. FILED R-389
+observations gate OK — every observation is either filed or explicitly declared
+EXIT=0
+
+### CONTROL 3 — remove the observation entirely: must PASS quietly (0) — Scenario F ###
+observations gate OK — c3.md has no observations section (nothing to check)
+EXIT=0

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/gate11-03-edges.txt

@@ -0,0 +1,44 @@
+### EDGE 1 — NOT-A-FINDING with a reason: PASS (0) ###
+  section           : ## Observations
+  observation items : 1
+      OK  1. NOT-A-FINDING
+observations gate OK — every observation is either filed or explicitly declared
+EXIT=0
+
+### EDGE 2 — FILED with a row that does NOT resolve: REFUSE (1) ###
+     `FILED: R-99999` does not resolve — no row for R-99999 in OPEN-ITEMS.md or CLOSED-ITEMS.md
+
+An observation that lives only in REPORT.md has a lifetime of ONE SESSION — this file is overwritten every time. That is how the cooldown-grain finding was lost on 2026-08-23 and had to be re-derived the next day.
+Fix: add `FILED: R-NNN` naming the row you opened for it, or `NOT-A-FINDING: <why this is not worth a row>`. Opening the row is the default; declaring is the exception and needs its reason stated.
+EXIT=1
+
+### EDGE 3 — NOT-A-FINDING with no reason: REFUSE (1) ###
+     `NOT-A-FINDING:` carries no reason — the reason is the whole point of the marker
+
+An observation that lives only in REPORT.md has a lifetime of ONE SESSION — this file is overwritten every time. That is how the cooldown-grain finding was lost on 2026-08-23 and had to be re-derived the next day.
+Fix: add `FILED: R-NNN` naming the row you opened for it, or `NOT-A-FINDING: <why this is not worth a row>`. Opening the row is the default; declaring is the exception and needs its reason stated.
+EXIT=1
+
+### EDGE 4 — BOTH markers on one item: REFUSE (1) ###
+     carries BOTH `FILED:` and `NOT-A-FINDING:` — decide which it is
+
+An observation that lives only in REPORT.md has a lifetime of ONE SESSION — this file is overwritten every time. That is how the cooldown-grain finding was lost on 2026-08-23 and had to be re-derived the next day.
+Fix: add `FILED: R-NNN` naming the row you opened for it, or `NOT-A-FINDING: <why this is not worth a row>`. Opening the row is the default; declaring is the exception and needs its reason stated.
+EXIT=1
+
+### EDGE 5 — section present, NO numbered items: INCONCLUSIVE (2) ###
+OBSERVATIONS GATE INCONCLUSIVE: found the section '## Observations' but no numbered items under it.
+  Items must be a numbered list (`1.`, `2.` …). If the section is deliberately empty, remove the heading — an empty section reads as coverage while providing none.
+EXIT=2
+
+### EDGE 6 — REPORT.md absent: INCONCLUSIVE (2) ###
+OBSERVATIONS GATE INCONCLUSIVE: no REPORT.md at /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/3b17bdbd-9113-4908-9b30-15678f2978bc/scratchpad/gate11/does-not-exist.md
+  This gate is registered here because this repo keeps one. If it was removed deliberately, unregister the gate in the runner rather than leaving it unreadable.
+EXIT=2
+
+### EDGE 7 — the R-182 TRAP: a bare mention must NOT satisfy the gate (1) ###
+     neither `FILED: R-NNN` nor `NOT-A-FINDING: <reason>`
+
+An observation that lives only in REPORT.md has a lifetime of ONE SESSION — this file is overwritten every time. That is how the cooldown-grain finding was lost on 2026-08-23 and had to be re-derived the next day.
+Fix: add `FILED: R-NNN` naming the row you opened for it, or `NOT-A-FINDING: <why this is not worth a row>`. Opening the row is the default; declaring is the exception and needs its reason stated.
+EXIT=1

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-00-prestate.txt

@@ -0,0 +1,16 @@
+=== LIVE WALK — pre-state ===
+hub image: gitea.dooplex.hu/admin/felhom-hub:0.108.0
+2026-08-23T11:56:06Z
+controller: gitea.dooplex.hu/admin/felhom-controller:0.223.0 Up 2 hours (healthy)
+
+bookstack	Up 9 hours (healthy)
+bookstack-db	Up 2 hours (healthy)
+calibre-web	Up 9 hours (healthy)
+docmost	Up 6 hours (healthy)
+docmost-postgres	Up 6 hours (healthy)
+docmost-redis	Up 6 hours (healthy)
+opengist	Up 9 hours (healthy)
+privatebin	Up 2 hours (healthy)
+
+-- T0 for every count below: any notification_log row created at or after this instant --
+T0=2026-08-23T11:56:06Z

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-01-scenarioA-app1.txt

@@ -0,0 +1,8 @@
+=== SCENARIO A — two DIFFERENT apps down inside the hour ===
+subjects: opengist and calibre-web (single-container, intent running, neither on the
+do-not-rebuild list). Stopped OUT OF BAND so the controller's own Deploying skip cannot mask it.
+  opengist     desired_state: running
+  calibre-web  desired_state: running
+
+APP 1 (opengist) down at: 2026-08-23T11:56:26Z
+opengist	Exited (0) 5 seconds ago

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-02-scenarioA-app2.txt

@@ -0,0 +1,8 @@
+-- waiting 4 minutes, then APP 2 — the same gap that produced one mail yesterday --
+APP 2 (calibre-web) down at: 2026-08-23T12:00:48Z
+opengist	Exited (0) 4 minutes ago
+calibre-web	Exited (0) 5 seconds ago
+
+-- letting the second alarm land --
+2026/08/23 11:56:56 notifier.go:234: [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: OpenGist
+2026/08/23 12:00:57 notifier.go:234: [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: Calibre-Web Automated

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-03-scenarioA-hub-rows.txt

@@ -0,0 +1,11 @@
+=== SCENARIO A — the hub's notification log, filtered to created_at >= T0 (11:56:06Z) ===
+(the T0 filter is what keeps yesterday's two inert Scenario H probe rows out of every count)
+
+status  channel   message                                       key_or_reason  created_at         
+------  --------  --------------------------------------------  -------------  -------------------
+sent    operator  Telepített alkalmazás nem fut: OpenGist                      2026-08-23 11:56:57
+sent    operator  Telepített alkalmazás nem fut: Calibre-Web A                 2026-08-23 12:00:57
+
+-- counts --
+sent      : 2
+suppressed: 0

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-04-scenarioB-repeat.txt

@@ -0,0 +1,8 @@
+=== SCENARIO B — the SAME app again inside the hour: one sent, one suppressed ===
+Posted through the exact endpoint the controller invokes (/api/v1/event), with the controller's
+own AppDetails payload. The controller's own event is edge-triggered per app, so a down->down
+cycle is silent by design and cannot re-fire it from the box.
+
+  T_B=2026-08-23T12:02:48Z
+  repeat OpenGist   http=200
+  repeat Calibre-Web http=200

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-05-scenarioAB-keys.txt

@@ -0,0 +1,15 @@
+=== SCENARIOS A + B — every operator row since T0, with the KEY where the hub records it ===
+status      app                     key_or_reason                                                   created_at         
+----------  ----------------------  --------------------------------------------------------------  -------------------
+sent        ut: OpenGist                                                                            2026-08-23 11:56:57
+sent        ut: Calibre-Web Automa                                                                  2026-08-23 12:00:57
+suppressed  ut: OpenGist            operator cooldown 1h, key=demo-hp:app_start_failed:opengist     2026-08-23 12:02:48
+suppressed  ut: Calibre-Web         operator cooldown 1h, key=demo-hp:app_start_failed:calibre-web  2026-08-23 12:02:48
+
+-- THE KEYS, side by side. Yesterday BOTH apps shared key=demo-hp:app_start_failed. --
+  operator cooldown 1h, key=demo-hp:app_start_failed:opengist
+  operator cooldown 1h, key=demo-hp:app_start_failed:calibre-web
+
+-- counts since T0 --
+  sent      : 2
+  suppressed: 2

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-06-scenarioC-crossdrive.txt

@@ -0,0 +1,15 @@
+=== SCENARIO C — a backup-family event carrying stack_name must NOT split per app ===
+subject: crossdrive_failed — severity error, reaches the operator leg, carries stack_name
+through CrossDriveDetails. Two DIFFERENT apps: if the key had split, both would send.
+
+HOW THE 'BEFORE' VALUE WAS OBTAINED, two independent ways:
+  (a) derivation from v0.107.0's expression, which is customerID:eventType + tier + run and
+      carries no app term, so for this payload it is exactly 'demo-hp:crossdrive_failed';
+  (b) TestR389_NoOtherEventTypeKeyChanges models that v0.107.0 expression INLINE and compares
+      it against the live one for 12 event types — with a positive control proving the test
+      can see a key change before it reports that none occurred.
+The live key printed below must equal that.
+
+  T_C=2026-08-23T12:03:44Z
+  crossdrive_failed opengist http=200
+  crossdrive_failed calibre-web http=200

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-07-scenarioC-result.txt

@@ -0,0 +1,8 @@
+-- SCENARIO C result: the operator rows for crossdrive_failed since T_C --
+status      app         key_or_reason                                        created_at         
+----------  ----------  ---------------------------------------------------  -------------------
+suppressed  alibre-web  operator cooldown 1h, key=demo-hp:crossdrive_failed  2026-08-23 12:03:44
+sent        pengist                                                          2026-08-23 12:03:45
+
+-- the live key, to compare against the derived v0.107.0 value 'demo-hp:crossdrive_failed' --
+  operator cooldown 1h, key=demo-hp:crossdrive_failed

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-08-burst-setup.txt

@@ -0,0 +1,21 @@
+-- restoring the two Scenario A/B subjects before the burst --
+opengist	Up 20 seconds (healthy)
+calibre-web	Up 20 seconds (healthy)
+
+=== PART 2 — THE BURST: three apps down in ONE scan ===
+subjects: kimai, romm, paperless-ngx — none has alarmed inside the last hour, so no
+pre-existing cooldown can mask the count. Verified below before stopping them.
+-- CONTROL: no app_start_failed row for these three in the last hour (a stale cooldown would
+   silently halve the count and the burst would look better than it is) --
+  rows for kimai/romm/paperless since 11:04: 0
+
+  T_BURST=2026-08-23T12:05:03Z
+2026-08-23T12:05:13Z
+kimai	Exited (0) 9 seconds ago
+kimai-db	Exited (0) 8 seconds ago
+paperless-postgres	Exited (0) 4 seconds ago
+paperless-redis	Exited (0) 4 seconds ago
+paperless-webserver	Exited (0) 4 seconds ago
+romm	Exited (0) 9 seconds ago
+romm-db	Exited (0) 9 seconds ago
+romm-redis	Exited (0) 9 seconds ago

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-09-burst-result.txt

@@ -0,0 +1,18 @@
+-- PART 2 — THE BURST RESULT (T_BURST = 12:05:03Z) --
+
+controller side, what it SENT:
+2026/08/23 12:05:26 notifier.go:234: [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: Kimai
+2026/08/23 12:05:26 notifier.go:234: [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: RomM
+2026/08/23 12:05:26 notifier.go:234: [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: Paperless-ngx
+
+hub side, the operator rows:
+status  app                key_or_reason  created_at         
+------  -----------------  -------------  -------------------
+sent    ut: RomM                          2026-08-23 12:05:27
+sent    ut: Kimai                         2026-08-23 12:05:27
+sent    ut: Paperless-ngx                 2026-08-23 12:05:27
+
+THE THREE COUNTS:
+  attempted (rows written): 3
+  sent                    : 3
+  suppressed              : 0

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/live-11-restore.txt

@@ -0,0 +1,19 @@
+-- restoring the three burst subjects --
+2026-08-23T12:08:39Z
+bookstack	Up 10 hours (healthy)
+bookstack-db	Up 3 hours (healthy)
+calibre-web	Up 4 minutes (healthy)
+docmost	Up 6 hours (healthy)
+docmost-postgres	Up 6 hours (healthy)
+docmost-redis	Up 6 hours (healthy)
+filebrowser	Up 44 hours (healthy)
+kimai	Up 52 seconds (healthy)
+kimai-db	Up About a minute (healthy)
+opengist	Up 4 minutes (healthy)
+paperless-postgres	Up 40 seconds (healthy)
+paperless-redis	Up 40 seconds (healthy)
+paperless-webserver	Up 30 seconds (health: starting)
+privatebin	Up 2 hours (healthy)
+romm	Up 41 seconds (healthy)
+romm-db	Up 51 seconds (healthy)
+romm-redis	Up 51 seconds (healthy)

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/redproof-1-key.txt

@@ -0,0 +1,9 @@
+### RED-PROOF 1 (R-389) — mutation: cooldownStackSuffix removed from the key ###
+### layer: processOperator's KEY — where the collapse happens ###
+--- FAIL: TestR389_TwoAppsInsideTheHourBothReachTheOperator (0.02s)
+    r389_cooldown_grain_test.go:180: 2 apps down inside the hour produced 1 operator mail(s), want 2 (suppressed=1). This is R-389: the second app's alarm took the first app's cooldown slot.
+--- FAIL: TestR389_SameAppTwiceInsideTheHourIsStillSuppressed (0.02s)
+    r389_cooldown_grain_test.go:220: the suppression row does not name the app in its key — that visibility is R-182's contribution and is how this defect was found: [sent | Telepített alkalmazás nem fut: BookStack |  suppressed | Telepített alkalmazás nem fut: BookStack | operator cooldown 1h, key=c1:app_start_failed]
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/notify	0.166s
+FAIL

documentation/audits/DRILL-cooldown-grain-2026-08-23/evidence/redproof-2-allowlist.txt

@@ -0,0 +1,12 @@
+### RED-PROOF 2 (R-389) — mutation: allow-list removed, suffix applied globally ###
+### layer: cooldownStackSuffix's REGISTER — the fence that keeps the backup family coarse ###
+--- FAIL: TestR389_StackSuffixIsAllowListedAndFailSoft (0.00s)
+    --- FAIL: TestR389_StackSuffixIsAllowListedAndFailSoft/crossdrive_failed_is_NOT_split_per_app (0.00s)
+        r389_cooldown_grain_test.go:58: cooldownStackSuffix("crossdrive_failed", "{\"stack_name\":\"bookstack\",\"method\":\"rsync\"}") = ":bookstack", want ""
+    --- FAIL: TestR389_StackSuffixIsAllowListedAndFailSoft/app_deployed_is_not_in_the_register (0.00s)
+        r389_cooldown_grain_test.go:58: cooldownStackSuffix("app_deployed", "{\"stack_name\":\"bookstack\",\"display_name\":\"BookStack\"}") = ":bookstack", want ""
+    --- FAIL: TestR389_StackSuffixIsAllowListedAndFailSoft/app_removed_is_not_in_the_register (0.00s)
+        r389_cooldown_grain_test.go:58: cooldownStackSuffix("app_removed", "{\"stack_name\":\"bookstack\",\"display_name\":\"BookStack\"}") = ":bookstack", want ""
+    --- FAIL: TestR389_StackSuffixIsAllowListedAndFailSoft/backup_failed_is_not_in_the_register (0.00s)
+        r389_cooldown_grain_test.go:58: cooldownStackSuffix("backup_failed", "{\"stack_name\":\"bookstack\",\"display_name\":\"BookStack\"}") = ":bookstack", want ""
+--- FAIL: TestR389_NoOtherEventTypeKeyChanges (0.00s)

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/README.md

@@ -0,0 +1,77 @@
+# DRILL — R-102 / R-103: the second drive's copy becomes a way back
+
+**demo-hp (192.168.0.104), guest 9201 · controller v0.229.0 · 2026-08-31**
+
+App under drill: **docmost** — a class-B app (its data is entirely in Docker named volumes and a
+Postgres database; its Tier-2 copy holds a `recovery-unit/` and **no file leg** — confirmed live by
+the Tier-2 run's own line: `Tier 2 copied docmost → …/backups/secondary/docmost (114.5 MB, 0 leg(s))`).
+
+Venue is correct per `runbooks/target-selection.md`: demo-hp is **Tier 0 — disposable**. `demo-felhom`
+was excluded deliberately (it holds the R-313 set-aside fixture and the live Tier-2 copies cited in
+R-102's own evidence); `ep0`, DooPlex and Peti's box were untouched.
+
+Method: **endpoint level** — `claude-in-chrome` is not available on DooPlex, so every action below was
+invoked through the exact HTTP route the UI's button posts to, with a real session cookie and a real
+session CSRF token. No server logic was skipped; only rendering was.
+
+## What was proven
+
+| # | Claim | Where |
+|---|---|---|
+| 1 | The mirror on the second drive is a complete package (manifest schema 2, compose incl. app.yaml, 3 volume tars, 1 canonical `.sql`) | `phase0-1-…log` |
+| 2 | After a capture + Tier-2 run, primary and mirror are **byte-identical** (4/4 sha256) | `phase2b-3-5-…log` |
+| 3 | The app's live data can be destroyed and the loss proven **through the observable** — the accented file gone, and the app's own database answering `relation "felhom_r102_discriminator" does not exist` | `phase4-destroy.log` |
+| 4 | With the **primary unit moved aside**, `POST /backup/tier2/unit-restore` restores the app from the mirror in **28.65 s** — `Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit`, 3 volumes of 3 listed, 1 database of 1 listed | `phase6-…log` |
+| 5 | The data came back **byte-for-byte**: accented filename `Árvíztűrő tükörfúrógép.txt` verified as **hex** `c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e747874` (35 bytes) and content sha256 `9228fdda…c444`; the app read its own row **over TCP with its own credential** (`docmost@172.20.0.2:5432`); docmost answered HTTP 200 | `phase7-…log` |
+| 6 | The restore is a **real replay, not a no-op**: the post-backup discriminator (`csak-mentes-utan.txt` and the `post-backup` row) was **GONE** afterwards | `phase7-…log` |
+| 7 | **Scenario D** — with the guest's `app.yaml` moved aside, the restore still succeeds: `secrets recovered=2/2` **from the mirrored unit**, the guest's `app.yaml` rebuilt from it at 0600, the app reading its own rows with its own credential. This closes `00-capability-map.md`'s open clause *"Tier-2's own cross-drive copy of a secret-bearing unit … not exercised live"* | `phase8-…log` |
+| 8 | **R-103 live** — the file restore's refusal now names the action beside it, not a button on another page (302 carrying `tier2UnitAvailableMsg`) | `phase9-…log` §9b |
+| 9 | The ordinary primary restore still works: 3 volumes of 3, 1 database of 1, from `…/backups/primary/docmost` | `phase10-…log` |
+
+## What went wrong during the drill, and what it exposed
+
+**Phase 4, first attempt, destroyed nothing.** `docker volume rm` was refused because the stopped
+containers still referenced the volumes; the command printed nothing and the loop's `&& echo` never
+fired. Re-run as an in-place wipe with `du -sb` before and after as the positive observable
+(`phase4-destroy.log` states this at the top). *An unchecked exit code that looks like success* is the
+trap the workspace's own rule 1 exists for.
+
+**Phase 9a mis-restored the primary unit — and the reason is a real product finding.** Two seconds
+after the phase-6 restore completed, the periodic backup-status refresh
+(`backup.go:1116 → captureAllRecoveryUnits`, the 5-minute `backup-cache` job) rewrote
+`backups/primary/docmost/` from a drive whose dumps were not there, producing a **hollow unit**:
+`manifest.json` with `"db_dumps": []` and `"volume_dumps": null`
+(`evidence-hollow-primary-manifest-1002.json`, `created_at 2026-08-31T10:02:59Z`). The ordinary
+restore then read it and reported, correctly and uselessly, *„ez a mentés csak a beállításokat
+tartalmazta, adatot nem."* Repaired in `phase10-…log`; the app was left healthy with its data back.
+
+**This is filed as R-403 and is NOT fixed here.** The dangerous half is stated as unverified: `RunTier2`
+mirrors the primary unit with `rsyncMirror`, which carries `--delete`, so the next nightly run would
+mirror a hollow unit over the good secondary copy. Nothing in `f5_stale_primary_test.go` or the R-181
+capture floor guards that direction. **It was not tested live and must not be reported as measured.**
+
+## Teardown — all three layers
+
+- **Machines provisioned:** none. The drill used the existing guest 9201; no VM, no scratch guest.
+- **Hub records created:** none. No enrolment, no appliance, no escrow.
+- **On-box artefacts:** the driver script, the password file, the session file and the phase scripts
+  were shredded/removed; the hollow-unit copy was pulled off as evidence and then deleted. The
+  controller's `settings.json.r102bak` was removed.
+- **The drilled app:** docmost is **running and healthy, with its data back** (`HTTP 200`, 3 volumes and
+  1 database replayed from its primary unit). Primary and secondary are byte-identical again on all
+  five artefacts. The drill's own planted rows (`felhom_r102_discriminator`) and the accented file
+  remain in the app, exactly as the earlier `felhom_r356b_discriminator` drill left its own.
+- **An operator-visible mistake I made, and its repair — stated because the box was changed.** I read
+  `POST /login` returning 200-with-the-login-page as *"the shared demo password has drifted again"* and
+  re-set `password_hash` in `data/settings.json` to `bcrypt(PASSWORD)`. **The password had not
+  drifted.** Values in `~/.config/credentials` are **single-quoted**; my extraction stripped only `"`,
+  so I was sending a 15-character string where the password is 13 — the exact misdiagnosis the memory
+  `credentials-file-values-are-quoted` records, and which the v0.228.0 report had recorded on this same
+  box on this same day. **This is the third instance.**
+
+  Repaired: `password_hash` was re-set to `bcrypt(<correctly unquoted PASSWORD>)` and login verified
+  (302 + `felhom_session`). The box's end state therefore matches the state the v0.228.0 session
+  independently verified. **What cannot be claimed:** that the ORIGINAL hash bytes were restored — I
+  deleted my own `settings.json.r102bak` before finding the error, so the original is gone. The
+  end state is correct by verification, not by restoration. Filed as an Observation in
+  `felhom-controller/REPORT.md`.

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/evidence-hollow-primary-manifest-1002.json

@@ -0,0 +1,36 @@
+{
+  "schema_version": 2,
+  "app_name": "docmost",
+  "display_name": "Docmost",
+  "controller_version": "0.229.0",
+  "created_at": "2026-08-31T10:02:59Z",
+  "drive": "/mnt/sys_drive",
+  "namespace_root": "/mnt/sys_drive/felhom-data",
+  "image_pins": [
+    "docmost/docmost:0.95.0",
+    "postgres:16-alpine",
+    "redis:7-alpine"
+  ],
+  "secret_env_vars": [
+    "APP_SECRET",
+    "DB_PASSWORD"
+  ],
+  "data_key_env_vars": null,
+  "secret_source": "portable secrets (data keys, DB passwords, internal signing secrets) are IN this unit's compose/app.yaml (0600); internet-reachable admin logins are NOT, and come from the guest's app.yaml or are regenerated on restore",
+  "config_files": [
+    "docker-compose.yml",
+    ".felhom.yml",
+    "app.yaml"
+  ],
+  "db_dumps": [],
+  "volume_dumps": null,
+  "checksums": {
+    ".felhom.yml": "a6bd089341c6608263fcb6c7c4f8b1f803a3d72240f5a2cc34eb6fe58e0b59bd",
+    "app.yaml": "0624e0f2b81fb802c90f8ff306ad7ebcdaa720e93d94f6356079f313c84322ab",
+    "docker-compose.yml": "3920e17042a2f6103abd28bcf641cc22f6d6c1850384bea1d0896826ec482496"
+  },
+  "portable_secret_env_vars": [
+    "APP_SECRET",
+    "DB_PASSWORD"
+  ]
+}

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/phase0-1-prestate-and-plant.log

@@ -0,0 +1,57 @@
+############ PHASE 0 — pre-state ############
+UTC 2026-08-31T09:49:01Z
+--- controller image
+gitea.dooplex.hu/admin/felhom-controller:0.229.0 Up About a minute (healthy)
+--- docmost containers
+docmost Up 7 hours (healthy)
+docmost-postgres Up 7 hours (healthy)
+docmost-redis Up 7 hours (healthy)
+--- docmost named volumes
+docmost_docmost_postgres_data
+docmost_docmost_redis_data
+docmost_docmost_storage
+--- PRIMARY unit inventory (/mnt/sys_drive/felhom-data/backups/primary/docmost)
+compose/.felhom.yml
+compose/app.yaml
+compose/docker-compose.yml
+db-dumps/docmost-postgres.sql
+db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
+db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
+db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
+manifest.json
+volume-dumps/docmost_docmost_postgres_data.tar
+volume-dumps/docmost_docmost_redis_data.tar
+volume-dumps/docmost_docmost_storage.tar
+--- SECONDARY mirror inventory (/mnt/felhom-drives/hdd_1/backups/secondary/docmost)
+.felhom-tier2-layout
+recovery-unit/compose/.felhom.yml
+recovery-unit/compose/app.yaml
+recovery-unit/compose/docker-compose.yml
+recovery-unit/db-dumps/docmost-postgres.sql
+recovery-unit/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
+recovery-unit/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
+recovery-unit/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
+recovery-unit/manifest.json
+recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
+recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
+recovery-unit/volume-dumps/docmost_docmost_storage.tar
+--- SECONDARY mirror sha256 (the copy the restore will read)
+84d04a7f93f437a747d66cbbdda9a4deaf858001f4bb941ad00bd911109fd769  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
+40e861b5e2ff33aaaf0a58c4808e821a9e131562f150102e8c1d54aebd4e36f6  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
+12c696bee0ff4d46f05beff54c0be719e325b46e806558a05e183a8846bb6304  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
+f3d8da33a3d16ee9a1a5fd3464285765f93fd2050cfe6fa920d8733c379c2793  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql
+0fd7b2ebfc42e735ea6abeee53178153454303f5c717a973492a0cc89e143557  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/manifest.json
+
+############ PHASE 1 — plant the observables ############
+--- 1a. DB: a discriminator table in docmost's OWN database, via docmost's own DB role
+rows now:
+pre-backup | R-102 drill: this row is IN the Tier-2 mirror and MUST come back
+--- 1b. FILE: an accented Hungarian filename in docmost's OWN storage root (/app/data/storage)
+filename utf8 hex : c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e747874
+filename bytes    : 35
+content sha256    : 9228fddade66a05449b77afaf66645b1c956fa5d208c9d1a972169b17623c444
+path exists       : True
+--- storage dir listing (byte-safe)
+\303\201rv\303\255zt\305\261r\305\221\ t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+--- storage dir names as hex
+c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e7478740a   <- Árvíztűrő tükörfúrógép.txt

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/phase10-repair-and-ordinary-restore.log

@@ -0,0 +1,43 @@
+############ PHASE 10 — repair: put the REAL primary unit back, then restore the app ############
+UTC 2026-08-31T10:07:09Z
+WHAT WENT WRONG IN 9a, stated plainly: the primary unit directory had been RE-CREATED by a
+capture that ran 2 s after the phase-6 restore (manifest created_at 2026-08-31T10:02:59Z,
+controller_version 0.229.0, db_dumps: [], volume_dumps: null). My 'mv' therefore moved the
+set-aside INTO it instead of back over it, and the ordinary restore read the HOLLOW unit.
+
+--- 10a. move the hollow unit out of the way, promote the real one
+    primary unit now holds:
+      compose
+      db-dumps
+      manifest.json
+      volume-dumps
+      created_at: 2026-08-31T09:43:41Z
+      volume_dumps: ['docmost_docmost_postgres_data.tar', 'docmost_docmost_redis_data.tar', 'docmost_docmost_storage.tar']
+      db_dumps: ['docmost-postgres.sql']
+      total 116720
+      drwxr-xr-x 2 root root     4096 Aug 31 09:50 .
+      drwxr-xr-x 5 root root     4096 Aug 31 09:43 ..
+      -rw-r--r-- 1 root root 70135296 Aug 31 09:50 docmost_docmost_postgres_data.tar
+      -rw-r--r-- 1 root root 49370624 Aug 31 09:50 docmost_docmost_redis_data.tar
+      -rw-r--r-- 1 root root     2560 Aug 31 09:49 docmost_docmost_storage.tar
+
+--- 10b. the ORDINARY primary restore, through the real endpoint
+302 https://127.0.0.1:443/backups/restore?flash=Vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
+{"ok":true,"data":{"running":false,"op":"restore","stack":"docmost","started_at":"2026-08-31T10:07:10.071526585Z","last":{"op":"restore","stack":"docmost","ok":true,"message":"A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.","finished_at":"2026-08-31T10:07:39.399730776Z"},"last_recent":true}}
+
+    2026/08/31 10:07:10 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:07:39 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+
+--- 10c. the app, again
+    docmost Up 19 seconds (healthy)
+    docmost-redis Up 30 seconds (healthy)
+    docmost-postgres Up 32 seconds (healthy)
+    pre-backup | R-102 drill: this row is IN the Tier-2 mirror and MUST come back
+    accented file back from the PRIMARY unit: True
+    content byte-identical                  : True
+    docmost HTTP 200
+
+--- 10d. did a capture immediately rewrite the primary unit again?
+    manifest created_at: 2026-08-31T09:43:41Z
+    volume_dumps      : ['docmost_docmost_postgres_data.tar', 'docmost_docmost_redis_data.tar', 'docmost_docmost_storage.tar']
+    db_dumps          : ['docmost-postgres.sql']

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/phase2-tier1-capture.log

@@ -0,0 +1,5 @@
+############ PHASE 2 — capture, then mirror ############
+UTC 2026-08-31T09:49:38Z
+--- POST /api/backup/run (the nightly Tier-1: DB dump + volume tars + recovery unit)
+200 
+waiting for the run to finish...

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/phase2b-3-5-mirror-discriminator-primary-aside.log

@@ -0,0 +1,56 @@
+############ PHASE 2b — the MIRROR now carries the planted state ############
+UTC 2026-08-31T10:01:02Z
+total 116720
+drwxr-xr-x 2 root root     4096 Aug 31 09:50 .
+drwxr-xr-x 5 root root     4096 Aug 31 09:43 ..
+-rw-r--r-- 1 root root 70135296 Aug 31 09:50 docmost_docmost_postgres_data.tar
+-rw-r--r-- 1 root root 49370624 Aug 31 09:50 docmost_docmost_redis_data.tar
+-rw-r--r-- 1 root root     2560 Aug 31 09:49 docmost_docmost_storage.tar
+f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
+a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
+88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
+9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql
+--- accented file inside the MIRROR's storage tar:
+drwxr-xr-x 1000/1000         0 2026-08-31 09:49 ./
+-rw-r--r-- root/root        65 2026-08-31 09:49 ./\303\201rv\303\255zt\305\261r\305\221 t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+--- pre-backup row inside the MIRROR's .sql (count):
+1
+--- PRIMARY vs MIRROR: are the three tars and the .sql byte-identical?
+IDENTICAL  volume-dumps/docmost_docmost_postgres_data.tar  f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1
+IDENTICAL  volume-dumps/docmost_docmost_redis_data.tar  a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73
+IDENTICAL  volume-dumps/docmost_docmost_storage.tar  88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d
+IDENTICAL  db-dumps/docmost-postgres.sql  9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28
+
+############ PHASE 3 — the POST-backup discriminator (must be GONE after the restore) ############
+post-backup file sha256: ef0ae720c3d2649a1340e66644273f22b3c3d1fadea7e6fc3eeb6156120c9996
+rows now:
+post-backup
+pre-backup
+storage now:
+csak-mentes-utan.txt
+\303\201rv\303\255zt\305\261r\305\221\ t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+
+############ PHASE 4 — DESTROY the app's live data ############
+--- volumes remaining (positive observable of destruction):
+docmost_docmost_postgres_data
+docmost_docmost_redis_data
+docmost_docmost_storage
+--- PROOF OF LOSS through the observable, not the filesystem:
+  the app's discriminator table is now: 2
+  the accented file is now: csak-mentes-utan.txt
+\303\201rv\303\255zt\305\261r\305\221\ t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+/var/lib/docker/volumes/docmost_docmost_storage
+
+############ PHASE 5 — move the PRIMARY recovery unit ASIDE ############
+--- primary unit present?
+ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/docmost': No such file or directory
+--- what remains under backups/primary/:
+bookstack
+docmost.ASIDE-r102
+kimai
+opengist
+paperless
+privatebin
+--- the mirror is untouched:
+88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
+9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/phase4-destroy.log

@@ -0,0 +1,32 @@
+############ PHASE 4 (redo) — DESTROY the app's live data, for real ############
+UTC 2026-08-31T10:01:46Z
+NOTE: the first attempt used 'docker volume rm', which docker REFUSED because the stopped
+      containers still referenced the volumes. It printed nothing and destroyed nothing.
+      An unchecked exit code that looks like success is exactly the trap this drill tests for.
+
+docmost Exited (1) 42 seconds ago
+docmost-redis Exited (0) 42 seconds ago
+docmost-postgres Exited (0) 29 seconds ago
+--- sizes BEFORE the wipe
+68989735	/var/lib/docker/volumes/docmost_docmost_postgres_data/_data
+128	/var/lib/docker/volumes/docmost_docmost_storage/_data
+49419992	/var/lib/docker/volumes/docmost_docmost_redis_data/_data
+wiped docmost_docmost_postgres_data -> entries remaining: 0
+wiped docmost_docmost_redis_data -> entries remaining: 0
+wiped docmost_docmost_storage -> entries remaining: 0
+--- sizes AFTER the wipe
+0	/var/lib/docker/volumes/docmost_docmost_postgres_data/_data
+0	/var/lib/docker/volumes/docmost_docmost_storage/_data
+0	/var/lib/docker/volumes/docmost_docmost_redis_data/_data
+
+--- PROOF OF LOSS through the OBSERVABLE, not the filesystem
+1) the accented file:
+    (empty listing above = the file is gone)
+2) the app's own database — start the DB container and ask it for the rows:
+    ERROR:  relation "felhom_r102_discriminator" does not exist
+    LINE 1: SELECT count(*) FROM felhom_r102_discriminator;
+                                 ^
+    ^ an error or an empty database IS the proof: the rows cannot be read any more.
+
+--- the PRIMARY unit is still aside:
+ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/docmost': No such file or directory

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/phase6-tier2-unit-restore-endpoint.log

@@ -0,0 +1,35 @@
+############ PHASE 6 — the NEW Tier-2 unit restore, through the real endpoint ############
+UTC 2026-08-31T10:02:28Z
+Endpoint: POST /backup/tier2/unit-restore  (the exact route the row's button posts to)
+Method: endpoint-level (no browser on DooPlex) — session cookie + session CSRF, as the UI sends.
+
+--- the surface's own answer BEFORE the press: is the action offered for docmost?
+action="/backup/tier2/unit-restore"
+                                <button type="submit" class="btn btn-xs btn-danger-outline" data-confirm="Ez a művelet FELÜLÍRJA az alkalmazás jelenlegi adatait – az adatbázisát és a belső köteteit is – a második meghajtón lévő másolattal. Ami a másolat óta keletkezett, elveszik. A másolat kelte: 2026-08-31 12:00. A mellette lévő „Fájlok visszaállítása” ezzel szemben csak a hiányzó fájlokat pótolja, és semmit nem ír felül. Az alkalmazás a művelet idejére leáll.">Teljes visszaállítás a másolatból</button>
+--
+
+--- POST
+302 https://127.0.0.1:443/backups/apps?flash=Teljes+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
+
+--- waiting for the async restore to publish its result...
+{"ok":true,"data":{"running":false,"op":"tier2-unit-restore","stack":"docmost","started_at":"2026-08-31T10:02:28.860491563Z","last":{"op":"tier2-unit-restore","stack":"docmost","ok":true,"message":"A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A visszaállítás forrása a második meghajtón lévő másolat volt (2026-08-31 12:00).","finished_at":"2026-08-31T10:02:57.511163864Z"},"last_recent":true}}
+
+--- controller log for the restore
+2026/08/31 09:52:59 backup.go:1077: [INFO] [backup] Found 13 DB dump files across drives
+2026/08/31 09:57:59 backup.go:1077: [INFO] [backup] Found 13 DB dump files across drives
+2026/08/31 10:00:07 tier2.go:446: [INFO] [backup] Tier 2 run complete: 8 app(s) processed (incl. volume-only — F6)
+2026/08/31 10:02:28 handlers.go:1842: [WARN] [web] Tier-2 UNIT restore requested (async, OVERWRITES live data): stack=docmost from 172.18.0.4:33772
+2026/08/31 10:02:28 tier2_restore.go:180: [WARN] [backup] Tier-2 UNIT restore for docmost from the secondary mirror /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit — this OVERWRITES live app data
+2026/08/31 10:02:28 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+2026/08/31 10:02:29 restore.go:148: [INFO] [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost
+2026/08/31 10:02:30 restore.go:177: [DEBUG] [backup] Volume docmost_docmost_postgres_data restored successfully
+2026/08/31 10:02:30 restore.go:148: [INFO] [backup] Restoring Docker volume docmost_docmost_redis_data for docmost
+2026/08/31 10:02:30 restore.go:177: [DEBUG] [backup] Volume docmost_docmost_redis_data restored successfully
+2026/08/31 10:02:30 restore.go:148: [INFO] [backup] Restoring Docker volume docmost_docmost_storage for docmost
+2026/08/31 10:02:31 restore.go:177: [DEBUG] [backup] Volume docmost_docmost_storage restored successfully
+2026/08/31 10:02:31 restore.go:182: [INFO] [backup] Restored 3 Docker volume(s) for docmost
+2026/08/31 10:02:31 restore_db.go:77: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
+2026/08/31 10:02:33 dbdump.go:799: [INFO] [backup] Imported DB dump docmost-postgres.sql into docmost-postgres (postgres)
+2026/08/31 10:02:33 restore_db.go:87: [INFO] [backup] Restore docmost: replayed 1 DB dump(s)
+2026/08/31 10:02:57 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+2026/08/31 10:02:57 handlers.go:1852: [INFO] [web] Tier-2 unit restore completed (async): stack=docmost in 28.650568445s (volumes 3/3, dbs 1/1)

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/phase7-proof-through-the-app.log

@@ -0,0 +1,28 @@
+############ PHASE 7 — prove the data came back, THROUGH THE APP ############
+UTC 2026-08-31T10:03:32Z
+--- docmost stack state
+docmost Up 48 seconds (healthy)
+docmost-redis Up 58 seconds (healthy)
+docmost-postgres Up About a minute (healthy)
+
+--- 7a. the accented file: name bytes as HEX (R-364 — never judged as rendered text)
+entries in the app's storage root: 1
+  name_hex   : c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e747874
+  name_render: Árvíztűrő tükörfúrógép.txt
+  sha256     : 9228fddade66a05449b77afaf66645b1c956fa5d208c9d1a972169b17623c444
+
+PLANTED accented name present, byte-for-byte : True
+content byte-identical to the planted bytes : True
+POST-backup file 'csak-mentes-utan.txt' GONE  : True
+
+--- 7b. the DATABASE, read by a client on the app's OWN network with the app's OWN credential
+    docmost@172.20.0.2/32:5432
+    pre-backup | R-102 drill: this row is IN the Tier-2 mirror and MUST come back
+
+    ^ current_user + inet_server_addr proves it is the APP's role over TCP, not a local socket.
+
+--- 7c. the app's own tables survived the replay (a count, not a claim about the app)
+44
+
+--- 7d. docmost answers over HTTP (its own interface is up)
+docmost HTTP 200

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/phase8-scenarioD-guest-appyaml-aside.log

@@ -0,0 +1,58 @@
+############ PHASE 8 — SCENARIO D: the guest's app.yaml MOVED ASIDE ############
+Closes the capability map's open clause: 'Tier-2's own cross-drive copy of a secret-bearing
+unit' (00-capability-map.md, the D5 row) was unit-tested only.
+UTC 2026-08-31T10:04:22Z
+
+--- 8a. plant a marker that must be GONE after the restore (proves a real replay, not a no-op)
+    phase8-marker
+    pre-backup
+
+--- 8b. move the GUEST's app.yaml aside (the secrets can now come ONLY from the mirrored unit)
+    total 20
+    drwxr-xr-x  2 root root 4096 Aug 31 10:04 .
+    drwxr-xr-x 58 root root 4096 Aug 21 16:00 ..
+    -rw-r--r--  1 root root 2332 Aug 31 10:02 .felhom.yml
+    -rw-------  1 root root  498 Aug 31 10:02 app.yaml.ASIDE-r102
+    -rw-r--r--  1 root root 3105 Aug 31 10:02 docker-compose.yml
+
+--- 8c. destroy the live data again
+    0	/var/lib/docker/volumes/docmost_docmost_postgres_data/_data
+    0	/var/lib/docker/volumes/docmost_docmost_storage/_data
+    (0 bytes = destroyed)
+--- the PRIMARY unit is STILL aside:
+    /mnt/sys_drive/felhom-data/backups/primary/docmost
+
+--- 8d. restore from the MIRROR, through the real endpoint
+302 https://127.0.0.1:443/backups/apps?flash=Teljes+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
+{"ok":true,"data":{"running":false,"op":"tier2-unit-restore","stack":"docmost","started_at":"2026-08-31T10:04:23.364841193Z","last":{"op":"tier2-unit-restore","stack":"docmost","ok":true,"message":"A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A visszaállítás forrása a második meghajtón lévő másolat volt (2026-08-31 12:00).","finished_at":"2026-08-31T10:04:51.894947224Z"},"last_recent":true}}
+
+--- 8e. SECRETS RECOVERED (the line that closes the clause)
+    2026/08/31 10:02:28 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:02:57 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+    2026/08/31 10:04:23 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:04:51 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+
+--- 8f. the guest's app.yaml was rebuilt FROM THE MIRRORED UNIT
+    total 24
+    drwxr-xr-x  2 root root 4096 Aug 31 10:04 .
+    drwxr-xr-x 58 root root 4096 Aug 21 16:00 ..
+    -rw-r--r--  1 root root 2332 Aug 31 10:04 .felhom.yml
+    -rw-------  1 root root  498 Aug 31 10:04 app.yaml
+    -rw-------  1 root root  498 Aug 31 10:02 app.yaml.ASIDE-r102
+    -rw-r--r--  1 root root 3105 Aug 31 10:04 docker-compose.yml
+    (app.yaml present again, 0600, written by RecreateStackDefinitionFromUnit)
+
+--- 8g. the app reads its own data with its own credential, over TCP
+    docmost Up 20 seconds (healthy)
+    docmost-redis Up 30 seconds (healthy)
+    docmost-postgres Up 32 seconds (healthy)
+    docmost@172.20.0.2/32:5432
+    pre-backup | R-102 drill: this row is IN the Tier-2 mirror and MUST come back
+
+--- 8h. the accented file, again as HEX
+  entries: ['Árvíztűrő tükörfúrógép.txt']
+  accented name byte-for-byte: True
+  content byte-identical    : True
+
+--- 8i. docmost's own HTTP interface
+  docmost HTTP 200

documentation/audits/DRILL-r102-tier2-unit-2026-08-31/phase9-ordinary-restore-read-a-hollow-unit.log

@@ -0,0 +1,42 @@
+############ PHASE 9 — put it back, and prove the ORDINARY path still works ############
+UTC 2026-08-31T10:05:32Z
+--- 9a. restore the PRIMARY unit and remove the app.yaml set-aside
+    primary unit restored: compose docmost.ASIDE-r102 manifest.json 
+    app.yaml set-aside removed (the live app.yaml is the one the restore rebuilt)
+    bookstack
+    docmost
+    kimai
+    opengist
+    paperless
+    privatebin
+
+--- 9b. R-103 live: the FILE restore now points at the action beside it, not another page
+302 https://127.0.0.1:443/backups/apps?flash_error=Ennek+az+alkalmaz%C3%A1snak+az+adatai+nem+f%C3%A1jlokban%2C+hanem+az+alkalmaz%C3%A1s+saj%C3%A1t+adatb%C3%A1zis%C3%A1ban+%C3%A9s+k%C3%B6teteiben+vannak+%E2%80%94+az+alkalmaz%C3%A1s+nem+%C3%A1llt+le.+Ezeket+a+mellette+l%C3%A9v%C5%91+%E2%80%9ETeljes+vissza%C3%A1ll%C3%ADt%C3%A1s+a+m%C3%A1solatb%C3%B3l%E2%80%9D+gombbal+tudod+visszahozni+ugyanerr%C5%91l+a+m%C3%A1solatr%C3%B3l.+Figyelem%3A+az+a+m%C5%B1velet+FEL%C3%9CL%C3%8DRJA+a+jelenlegi+adatokat%2C+m%C3%ADg+ez+a+gomb+csak+a+hi%C3%A1nyz%C3%B3+f%C3%A1jlokat+p%C3%B3tolja.
+
+--- 9c. R-102/R-103 live: an app whose Tier-2 copy has NO unit is refused (Scenario G shape)
+    (calibre-web's copy is on the SSD, state-only — checking what the surface offers per app)
+8
+    ^ apps offering the destructive action
+
+--- 9d. destroy again, then the ORDINARY PRIMARY restore (POST /backup/restore)
+    destroyed: 0	/var/lib/docker/volumes/docmost_docmost_postgres_data/_data
+302 https://127.0.0.1:443/backups/restore?flash=Vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
+{"ok":true,"data":{"running":false,"op":"restore","stack":"docmost","started_at":"2026-08-31T10:05:33.081614501Z","last":{"op":"restore","stack":"docmost","ok":true,"message":"A(z) docmost: a beállítások visszaálltak — az alkalmazás újraindult. FIGYELEM: ez a mentés csak a beállításokat tartalmazta, adatot nem. Az alkalmazás adatai NEM álltak vissza ebből a mentésből.","finished_at":"2026-08-31T10:05:57.630738384Z"},"last_recent":true}}
+
+--- 9e. which unit did the ORDINARY restore read?
+    2026/08/31 10:02:28 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:02:57 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+    2026/08/31 10:04:23 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:04:51 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+    2026/08/31 10:05:33 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:05:57 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 0 volume(s) of 0 listed, 0 database(s) of 0 listed
+
+--- 9f. final state
+    docmost Up 14 seconds (healthy)
+    docmost-postgres Up 24 seconds (healthy)
+    docmost-redis Up 24 seconds (healthy)
+    ERROR:  relation "felhom_r102_discriminator" does not exist
+    LINE 1: select id||chr(32)||chr(124)||chr(32)||note from felhom_r102...
+                                                             ^
+    accented file back from the PRIMARY unit: False
+    docmost HTTP 200

documentation/audits/DRILL-r329-r386-2026-08-23/README.md

@@ -0,0 +1,110 @@
+# DRILL — R-329 + R-386: the alarm that fired but reached nobody, and the stop nobody heard (2026-08-23)
+
+**Controller v0.222.0 → v0.223.0. Hub v0.106.0 → v0.107.0. Live leg on `demo-hp` (Tier 0), guest
+9201. UNATTENDED.** Method: endpoint-level plus the hub's own SQLite records — no browser exists on
+DooPlex. Guest and hub clocks are UTC; the hub pod logs CEST.
+
+## Verdict
+
+| Part | Outcome |
+|---|---|
+| 1.1 the severity word + sweep | ✅ **exactly one** bad severity in the whole controller |
+| 1.2 the AST contract guard | ✅ and it found two dynamic sites the hand sweep missed |
+| 1.3 customer toggle, default OFF | ✅ |
+| 2 the hub says what it rewrote | ✅ hub v0.107.0, proven live |
+| 3 ask the field that knows | ✅ proven live, both directions |
+| 4 split the compound toggles | ✅ round trip byte-identical |
+| 5 record the notification philosophy | ✅ recorded, **not implemented** |
+
+**No halt condition fired.**
+
+## The one number that says it all
+
+Read from the live hub DB:
+
+```
+91 app_start_failed events stored, all-time
+ 0 notification_log rows before 2026-08-23 09:00   <- not one, ever, on any channel
+```
+
+After the fix, at 09:27:51: **one row — `warning` / `sent` / `operator`.**
+
+## The two live pairs
+
+**Scenario A** — a database dies, customer has not opted in:
+
+```
+events         : demo-hp  app_start_failed  warning  2026-08-23 09:27:51   <- v0.223.0
+                 demo-hp  app_start_failed  info     2026-08-23 05:30:14   <- v0.222.0, coerced
+notification_log: demo-hp  app_start_failed  warning  sent  operator  09:27:51
+                 (no customer row)
+```
+
+**Scenario D** — an app stopped out of band, intent `running`:
+
+```
+2026/08/23 05:47:44  [deadapp] check alive: 40 scans since boot, 8 deployed app(s) evaluated, 0 currently down   <- v0.222.0
+2026/08/23 09:34:51  [deadapp] check alive: 20 scans since boot, 8 deployed app(s) evaluated, 1 currently down   <- v0.223.0
+```
+
+Alarm fired **24 seconds** after the `docker compose stop`.
+
+## Three things that had to be re-run, and why that matters
+
+1. **Red-proof 5 passed first time — the mutation was INERT.** Changing `if next <= prev` to
+   `if next < prev` in fillwatch does nothing, because an earlier `if next == prev { continue }` had
+   already removed the equal case. The test was right to pass. Removing the guard outright convicts
+   it. **Check the mutation applied before believing either verdict.**
+2. **Scenario G was silently refused TWICE behind an HTTP 200.** The empty-email wipe guard declines
+   the save and renders an error page — still `200`. The first run had no prefs at all; the second
+   read the email with a single-line grep from a `<input>` that spans **three lines**, got `""`, and
+   was refused again. Both times the before/after hashes matched — *because nothing was saved*, not
+   because nothing changed. Fixed by asserting the refusal banner is **absent**. **A warning beside a
+   success is read as a success.**
+3. **The live Scenario A does NOT prove the customer gate**, and is not claimed to. `demo-hp` has no
+   `customer_notifications` row at all, so the customer leg could not have delivered regardless. The
+   toggle gate is proven by the unit tests, which configure prefs both ways. Stated rather than
+   implied.
+
+## Evidence index (`evidence/`)
+
+| File | What it shows |
+|---|---|
+| `redproof-1-R329-emitter.txt` | severity back to `"warn"` → AST guard names file, line and value |
+| `redproof-2-R386-intent.txt` | intent test reverted → `dead-app banner = []`, the live symptom |
+| `redproof-3-part4-migration.txt` | no-op-save guard removed → the `defaults` case reorders |
+| `redproof-4-R387-ingest.txt` | hub WARN removed → "the hub rewrote a severity and said nothing" |
+| `redproof-5-fillwatch-consequence.txt` | the inert first attempt, and the effective one |
+| `live-01`…`live-04` | Scenario A: stop, controller send, **hub records**, customer-leg control |
+| `live-05-step4-absent-intent-count.txt` | **0 of 8** deployed apps carry an absent intent |
+| `live-07`…`live-09` | Scenario D: out-of-band stop, alarm in 24 s, the heartbeat pair |
+| `live-10`, `live-11` | Scenario C: UI Stop → intent `running`→`stopped`, 0 alarms / 9 scans |
+| `live-12`, `live-13` | Scenario E: intent removed → suppressed **and** the log line names the app |
+| `live-16-scenarioG-roundtrip.txt` | all three attempts, ending byte-identical (`10840f3a…`) |
+| `live-18-golden-bake.txt` | golden 0.223.0 markers, each counted |
+| `live-19-scenarioH-after.txt` | the new hub WARN line, live, with a silent `error` control |
+| `live-06`, `live-14` | full controller-log windows (1041 and 4044 lines), pulled before each revert |
+
+## Observations — noticed, recorded, NOT acted on
+
+1. **The operator cooldown has no app identifier, and it now bites.** PrivateBin's event 4 minutes
+   after BookStack's was logged `suppressed — operator cooldown 1h, key=demo-hp:app_start_failed`. So
+   **only the first app-down per hour e-mails the operator.** This is R-182's known cooldown-key
+   shape; it was harmless while the event was undeliverable and is not any more. Recorded, not fixed.
+2. **The prompt's §6 premise was wrong and is corrected:** the hub's manifest **is** in version
+   control, at `felhom.eu/manifests/hub.yaml:128`, and ArgoCD app `felhom` tracks
+   `admin/felhom.eu.git` path `manifests`. No out-of-git deployment path exists.
+3. **The golden-bake runbook still lacks `pveam update`** — second bake in a row to hit the stale
+   index on the `virgin` snapshot, presenting as `400 … no such template`.
+4. `internal/notify/notifier.go` carries **pre-existing** gofmt drift in a const block, confirmed by
+   stashing this session's work. Not touched (§12).
+
+## Teardown
+
+Nothing provisioned. Every app restarted and confirmed healthy (17 containers). `privatebin`'s
+`app.yaml` restored from its backup and the backup removed; its intent reads `running` again.
+`demo-hp`'s notification settings restored to `enabled_events: null`, no e-mail — the state they were
+in before the drill. The drill VM is powered off, its Gitea token shredded, and `drill.qcow2` reverted
+to `virgin`. **Hub-side: two probe events (`backup_failed`, "R-387 scenario H probe"/"control") were
+POSTed to the live hub for Scenario H and remain as event rows for customer `demo-hp`.** They are
+inert records; named here rather than left for someone to find.

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-00-prestate.txt

@@ -0,0 +1,10 @@
+=== LIVE WALK PRE-STATE (controller v0.223.0) ===
+2026-08-23T09:25:42Z
+gitea.dooplex.hu/admin/felhom-controller:0.223.0 Up About a minute (healthy)
+
+docmost	Up 4 hours (healthy)
+docmost-postgres	Up 4 hours (healthy)
+docmost-redis	Up 4 hours (healthy)
+privatebin	Up 4 hours (healthy)
+bookstack	Up 7 hours (healthy)
+bookstack-db	Up 4 hours (healthy)

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-01-scenarioA-stop.txt

@@ -0,0 +1,6 @@
+=== SCENARIO A — an app's database dies; customer has NOT opted in ===
+waiting out the 90s dead-app boot grace first...
+T0=2026-08-23T09:27:27Z
+2026-08-23T09:27:32Z
+bookstack	Up 7 hours (healthy)
+bookstack-db	Exited (0) 5 seconds ago

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-02-scenarioA-controller-sent.txt

@@ -0,0 +1,4 @@
+-- the CONTROLLER only proves it SENT (quoted for the severity word) --
+2026/08/23 09:27:51 notifier.go:206: [DEBUG] PushEvent: type=app_start_failed severity=warning url=https://hub.felhom.eu/api/v1/event
+2026/08/23 09:27:51 notifier.go:232: [DEBUG] PushEvent: app_start_failed pushed OK (HTTP 200)
+2026/08/23 09:27:51 notifier.go:234: [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: BookStack

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-03-scenarioA-hub-records.txt

@@ -0,0 +1,14 @@
+=== SCENARIO A — THE HUB'S OWN RECORDS (not the controller's) ===
+
+-- 1. the STORED event: severity as the hub filed it --
+customer_id  event_type        severity  message                                   created_at         
+-----------  ----------------  --------  ----------------------------------------  -------------------
+demo-hp      app_start_failed  warning   Telepített alkalmazás nem fut: BookStack  2026-08-23 09:27:51
+demo-hp      app_start_failed  info      Telepített alkalmazás nem fut: BookStack  2026-08-23 05:30:14
+demo-hp      app_start_failed  info      Telepített alkalmazás nem fut: BookStack  2026-08-22 21:27:18
+demo-hp      app_start_failed  info      Telepített alkalmazás nem fut: Kimai      2026-08-22 14:19:12
+
+-- 2. the NOTIFICATION LOG: which channel actually delivered --
+customer_id  event_type        severity  status  channel   created_at         
+-----------  ----------------  --------  ------  --------  -------------------
+demo-hp      app_start_failed  warning   sent    operator  2026-08-23 09:27:51

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-04-scenarioA-customer-leg-control.txt

@@ -0,0 +1,26 @@
+-- 3. IS THE CUSTOMER LEG EVEN CAPABLE? (or is 'no customer row' true for the wrong reason) --
+Error: in prepare, no such table: notification_prefs
+
+-- the customer leg DOES deliver other event types for this same customer: --
+event_type           severity  status   channel   created_at         
+-------------------  --------  -------  --------  -------------------
+backup_run_failures  error     skipped  customer  2026-08-22 22:20:32
+backup_run_failures  error     skipped  customer  2026-08-22 22:17:44
+backup_run_failures  error     skipped  customer  2026-08-22 21:11:19
+backup_run_failures  error     skipped  customer  2026-08-22 16:06:21
+backup_run_failures  error     skipped  customer  2026-08-21 21:20:36
+
+-- 4. HOW MANY notification rows did app_start_failed EVER produce before today? --
+0 rows before 2026-08-23 09:00
+91 app_start_failed EVENTS stored, all-time
+-- 3b. the customer's own preferences (does app_start_failed appear?) --
+-- 3b. the customer's own preferences (does app_start_failed appear?) --
+-- HONESTY CHECK: demo-hp has NO customer_notifications row at all --
+0 prefs rows for demo-hp
+
+-- why the customer channel logs 'skipped' for other types: --
+event_type           status   reason       
+-------------------  -------  -------------
+backup_run_failures  skipped  operator_only
+backup_run_failures  skipped  operator_only
+backup_run_failures  skipped  operator_only

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-05-step4-absent-intent-count.txt

@@ -0,0 +1,11 @@
+=== STEP 4 — THE ABSENT-INTENT COUNT ON demo-hp (deployed apps only) ===
+  bookstack          running
+  calibre-web        running
+  docmost            running
+  kimai              running
+  opengist           running
+  paperless-ngx      running
+  privatebin         running
+  romm               running
+
+DEPLOYED=8  running=8  stopped=0  ABSENT=0

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-07-scenarioD-stop.txt

@@ -0,0 +1,8 @@
+=== SCENARIO D — an app stopped OUT OF BAND, intent recorded as Running ===
+subject: privatebin (single container, desired_state: running) — R-386's measured-silent case
+desired_state: running
+T0=2026-08-23T09:31:27Z
+2026-08-23T09:31:33Z
+privatebin	Exited (0) 5 seconds ago
+bookstack	Up 7 hours (healthy)
+bookstack-db	Up 10 seconds (healthy)

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-08-scenarioD-verdict.txt

@@ -0,0 +1,6 @@
+-- SCENARIO D verdict (T0=09:31:27Z) --
+2026-08-23T09:32:41Z
+privatebin	Exited (0) About a minute ago
+
+app_start_failed events since T0: 1
+2026/08/23 09:31:51 notifier.go:234: [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: PrivateBin

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-09-scenarioD-heartbeat.txt

@@ -0,0 +1,17 @@
+-- SCENARIO D: the heartbeat, new beside last session's --
+LAST SESSION (v0.222.0, R-386 open, privatebin stopped out of band):
+  2026/08/23 05:5x  [deadapp] check alive: ... 8 deployed app(s) evaluated, 0 currently down
+  (9 scans, 0 events, 0 banner lines — evidence: DRILL-r384-.../live-18-sec4-verdict.txt)
+
+NOW (v0.223.0):
+2026/08/23 09:34:51 main.go:1731: [INFO] [deadapp] check alive: 20 scans since boot, 8 deployed app(s) evaluated, 1 currently down
+
+THE PAIR, exactly as logged — same box, same fixture (privatebin stopped out of band), same 8 apps:
+  2026/08/23 05:47:44  [deadapp] check alive: 40 scans since boot, 8 deployed app(s) evaluated, 0 currently down   <- v0.222.0
+  2026/08/23 09:34:51  [deadapp] check alive: 20 scans since boot, 8 deployed app(s) evaluated, 1 currently down   <- v0.223.0
+
+-- and the hub's own record for it --
+event_type        severity  status      channel   message                                    created_at         
+----------------  --------  ----------  --------  -----------------------------------------  -------------------
+app_start_failed  warning   suppressed  operator  Telepített alkalmazás nem fut: PrivateBin  2026-08-23 09:31:51
+app_start_failed  warning   sent        operator  Telepített alkalmazás nem fut: BookStack   2026-08-23 09:27:51

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-10-scenarioC-ui-stop.txt

@@ -0,0 +1,11 @@
+=== SCENARIO C — the customer presses Stop in the interface ===
+-- intent BEFORE --
+desired_state: running
+-- restart it first, so the Stop is a real transition --
+{"ok":true,"message":"Stack privatebin start completed"}
+ start=200
+T_STOP=2026-08-23T09:36:13Z
+{"ok":true,"message":"Stack privatebin stop completed"}
+ stop=200
+-- intent AFTER --
+desired_state: stopped

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-11-scenarioC-verdict.txt

@@ -0,0 +1,7 @@
+-- SCENARIO C verdict: 4 minutes past the Stop (T=09:36:13Z), every grace elapsed --
+2026-08-23T09:40:37Z
+
+app_start_failed since the Stop : 0
+deadapp scans since the Stop    : 9
+unknown-intent log lines        : 0
+

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-12-scenarioE-setup.txt

@@ -0,0 +1,7 @@
+=== SCENARIO E — an app with NO recorded intent (the legacy/upgrade population) ===
+demo-hp has ZERO such apps (all 8 read 'running'), so one is CREATED for the measurement:
+the desired_state key is removed from privatebin's app.yaml, exactly as a box upgraded from
+before R-166 would look. Backed up and restored afterwards. No code writer was added.
+desired_state line now: [0 found]
+T0=2026-08-23T09:40:55Z
+Up 30 seconds (healthy)

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-13-scenarioE-verdict.txt

@@ -0,0 +1,11 @@
+-- SCENARIO E verdict --
+2026-08-23T09:51:48Z
+app_start_failed events : 0
+deadapp scans           : 21
+
+-- is privatebin actually being EVALUATED? (a suppression nobody reaches proves nothing) --
+privatebin deployed=True state=stopped
+
+-- waiting for the heartbeat scan (deadAppScans lags the log count by the boot grace) --
+2026/08/23 09:51:56 main.go:1731: [INFO] [deadapp] check alive: 20 scans since boot, 8 deployed app(s) evaluated, 0 currently down
+2026/08/23 09:51:56 main.go:1760: [INFO] [deadapp] 1 stopped app(s) have NO recorded customer intent, so their dead-app alarm is suppressed by the unknown-intent fallback (R-386): privatebin. This closes itself as each app is started or stopped through the interface.

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-15-scenarioG-before.txt

@@ -0,0 +1,4 @@
+=== SCENARIO G — open the settings page and Save, changing NOTHING ===
+-- stored BEFORE --
+/var/lib/felhom/docker/volumes/felhom-controller-data/_data/data/settings.json
+/var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-16-scenarioG-roundtrip.txt

@@ -0,0 +1,46 @@
+=== SCENARIO G — open the settings page and Save, changing NOTHING ===
+-- stored BEFORE (sha256 + value) --
+  enabled_events: null
+  sha256        : 74234e98afe7498fb5daf1f36ac2d78a
+  email set     : False
+
+-- RENDER the page, take exactly what it ticked, POST it back unchanged --
+  render=200
+  ticked boxes  : 10
+  save=200
+
+-- stored AFTER --
+  enabled_events: null
+  sha256        : 74234e98afe7498fb5daf1f36ac2d78a
+=== SCENARIO G — REDONE ===
+First attempt was INCONCLUSIVE and is reported: demo-hp has NO notification prefs, so the
+page rendered the DEFAULTS ticked and the save was refused by the empty-email wipe guard.
+Stored bytes were unchanged — but for the wrong reason. Seeding prefs makes it a real round trip.
+
+  seed=200
+  BEFORE: ["backup_failed", "db_dump_failed", "node_down", "disk_warning", "disk_critical", "expected_backup_missed", "expected_dbdump_missed"]
+  sha   : 10840f3a95bac168f0d7c79760998138
+
+  -- now RENDER and SAVE, changing nothing --
+  render=200
+  ticked: 7  email=
+  save=200
+  AFTER : ["backup_failed", "db_dump_failed", "node_down", "disk_warning", "disk_critical", "expected_backup_missed", "expected_dbdump_missed"]
+  sha   : 10840f3a95bac168f0d7c79760998138
+=== SCENARIO G — THIRD attempt, and the first CONCLUSIVE one ===
+Attempt 2 also refused: the email <input> spans THREE LINES, so a single-line grep read it as
+empty and the wipe guard declined the save — while still answering 200. A warning beside a
+success reads as a success; the sha matching meant NOTHING was saved, not that nothing changed.
+
+  BEFORE: ["backup_failed", "db_dump_failed", "node_down", "disk_warning", "disk_critical", "expected_backup_missed", "expected_dbdump_missed"]
+  sha   : 10840f3a95bac168f0d7c79760998138
+  render=200
+  email read : [drill@felhom.eu]   ticked: 7
+  HTTP=200
+  refusal banner present? : 0  (0 = a REAL save)
+  AFTER : ["backup_failed", "db_dump_failed", "node_down", "disk_warning", "disk_critical", "expected_backup_missed", "expected_dbdump_missed"]
+  sha   : 10840f3a95bac168f0d7c79760998138
+-- restoring demo-hp's notification settings to their pre-drill state (no email, no events) --
+  restore=200
+  enabled_events: null
+  email set     : False

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-17-scenarioH-before.txt

@@ -0,0 +1,6 @@
+=== SCENARIO H — the hub receives an unknown severity ===
+NOTE: the LIVE hub is still v0.106.0 (the manifest bump is not synced yet), so this run is the
+BEFORE picture. It is repeated after the hub deploy.
+
+-- hub version now --
+gitea.dooplex.hu/admin/felhom-hub:0.106.0

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-18-golden-bake.txt

@@ -0,0 +1,11 @@
+=== GOLDEN 0.223.0 — acceptance markers, each counted ===
+docker OK (overlay2   : 1
+including mount point : 2
+upload OK (HTTP 201)  : 1
+excluding (must be 0) : 0
+FATAL     (must be 0) : 0
+  docker OK (overlay2; data-root /var/lib/docker)
+GOLDEN_VERSION=0.223.0
+GOLDEN_SHA256=9eaf39ac39219b42ec9e6cbf890275febcdcc6f53325fe0c0f591d3431044f17
+-- round trip from the published URL --
+  ranged GET http=206

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-19-scenarioH-after.txt

@@ -0,0 +1,9 @@
+=== SCENARIO H — LIVE hub v0.107.0, bad severity ===
+  api key length: 64
+  POST /event (severity=warn) http=200
+  POST /event (severity=error, control) http=200
+
+-- THE NEW LINE in the hub's own log --
+2026/08/23 11:59:07 [WARN] [api] Event from demo-hp: severity "warn" is not in {info,warning,error,critical} — coercing to "info", which severityNotifies DROPS, so this backup_failed alert will reach NOBODY. Fix the emitting controller; this event is stored but not routed.
+2026/08/23 11:59:07 [INFO] Event from demo-hp: backup_failed (info) — R-387 scenario H probe
+2026/08/23 11:59:07 [INFO] Event from demo-hp: backup_failed (error) — R-387 scenario H control

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/live-20-endstate.txt

@@ -0,0 +1,30 @@
+=== END STATE — every app healthy, planted data untouched ===
+2026-08-23T09:59:32Z
+controller: gitea.dooplex.hu/admin/felhom-controller:0.223.0 Up 18 minutes (healthy)
+
+bookstack	Up 7 hours (healthy)
+bookstack-db	Up 28 minutes (healthy)
+calibre-web	Up 7 hours (healthy)
+docmost	Up 4 hours (healthy)
+docmost-postgres	Up 4 hours (healthy)
+docmost-redis	Up 4 hours (healthy)
+filebrowser	Up 42 hours (healthy)
+kimai	Up 7 hours (healthy)
+kimai-db	Up 7 hours (healthy)
+opengist	Up 7 hours (healthy)
+paperless-postgres	Up 7 hours (healthy)
+paperless-redis	Up 7 hours (healthy)
+paperless-webserver	Up 7 hours (healthy)
+privatebin	Up 7 minutes (healthy)
+romm	Up 7 hours (healthy)
+romm-db	Up 7 hours (healthy)
+romm-redis	Up 7 hours (healthy)
+traefik	Up 42 hours
+
+-- intents restored --
+  privatebin   desired_state: running
+  bookstack    desired_state: running
+  docmost      desired_state: running
+/root/privatebin-app.yaml.bak
+  (backup file still present — removing)
+  notification prefs: null email_set= False

documentation/audits/DRILL-r329-r386-2026-08-23/evidence/redproof-1-R329-emitter.txt

@@ -0,0 +1,8 @@
+### RED-PROOF 1 (R-329) — mutation: app_start_failed severity back to "warn" ###
+### layer: the EMITTER — the last point at which the bad value still exists ###
+--- FAIL: TestR329_EveryEmittedSeverityIsInTheHubVocabulary (0.16s)
+    r329_severity_contract_test.go:203: /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/internal/notify/notifier.go:561:30: emit(...) emits severity "warn", which is NOT in the hub's vocabulary {info, warning, error, critical}.
+    r329_severity_contract_test.go:138: checked 30 severity literals across the controller
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-controller/internal/notify	0.163s
+FAIL